From fa23801996a67ef42542a001bcbf684860f2ce35 Mon Sep 17 00:00:00 2001 From: Kazuto Iris <78157415+kazutoiris@users.noreply.github.com> Date: Sat, 25 Oct 2025 12:20:00 -0400 Subject: [PATCH 1/2] docs(ecm): add documentation for ECM service - Add detailed documentation for ECM (Engine Connection Manager) service - Include API interfaces, database schemas, and RPC methods for ECM - Add comprehensive documentation for computation governance services - Create overview documentation for all service modules - Include RPC development examples and service structure information --- * update func md (aiceflower) * add jobhistory func (aiceflower) * add module function doc (aiceflower) Co-authored-by: aiceflower --- .gitignore | 5 +- docs/modules/README.md | 60 + docs/modules/computation-governance/README.md | 157 ++ docs/modules/computation-governance/ecm.md | 635 ++++++ .../computation-governance/entrance.md | 726 ++++++ .../computation-governance/jobhistory.md | 481 ++++ .../modules/computation-governance/manager.md | 1010 ++++++++ docs/modules/index.md | 40 + .../modules/microservice-governance/README.md | 112 + .../modules/microservice-governance/eureka.md | 131 ++ .../microservice-governance/gateway.md | 421 ++++ .../microservice-governance/monitor.md | 261 +++ docs/modules/public-enhancements/README.md | 235 ++ docs/modules/public-enhancements/bml.md | 634 +++++ .../public-enhancements/configuration.md | 1264 ++++++++++ docs/modules/public-enhancements/context.md | 962 ++++++++ .../modules/public-enhancements/datasource.md | 2032 +++++++++++++++++ .../modules/public-enhancements/jobhistory.md | 579 +++++ .../public-enhancements/publicservice.md | 151 ++ 19 files changed, 9895 insertions(+), 1 deletion(-) create mode 100644 docs/modules/README.md create mode 100644 docs/modules/computation-governance/README.md create mode 100644 docs/modules/computation-governance/ecm.md create mode 100644 docs/modules/computation-governance/entrance.md create mode 100644 docs/modules/computation-governance/jobhistory.md create mode 100644 docs/modules/computation-governance/manager.md create mode 100644 docs/modules/index.md create mode 100644 docs/modules/microservice-governance/README.md create mode 100644 docs/modules/microservice-governance/eureka.md create mode 100644 docs/modules/microservice-governance/gateway.md create mode 100644 docs/modules/microservice-governance/monitor.md create mode 100644 docs/modules/public-enhancements/README.md create mode 100644 docs/modules/public-enhancements/bml.md create mode 100644 docs/modules/public-enhancements/configuration.md create mode 100644 docs/modules/public-enhancements/context.md create mode 100644 docs/modules/public-enhancements/datasource.md create mode 100644 docs/modules/public-enhancements/jobhistory.md create mode 100644 docs/modules/public-enhancements/publicservice.md diff --git a/.gitignore b/.gitignore index 8610a1edd2..c6d34b7301 100644 --- a/.gitignore +++ b/.gitignore @@ -33,4 +33,7 @@ target/ # log folder *.log logs/ -nohup.out \ No newline at end of file +nohup.out + +# for claude +.claude \ No newline at end of file diff --git a/docs/modules/README.md b/docs/modules/README.md new file mode 100644 index 0000000000..27ce6a6f7d --- /dev/null +++ b/docs/modules/README.md @@ -0,0 +1,60 @@ +# Apache Linkis Service Modules + +## Overview + +Apache Linkis is built using a microservices architecture and can be categorized into three service types: + +1. **Microservice Governance Services**: Infrastructure services including service discovery, API gateway, and service communication +2. **Computation Governance Services**: Core services for computation task lifecycle management +3. **Public Enhancement Services**: Shared services that provide common capabilities across the platform + +Each service type contains multiple actual deployable Java applications (sub-modules) that work together to provide the complete functionality. + +## Service Structure + +Based on the actual deployment structure, the services are organized as follows: + +### Microservice Governance Services +These are the infrastructure services that provide the foundation for the microservices architecture: +- [Eureka Service](./microservice-governance/eureka.md) - Service registry and discovery +- [Gateway Service](./microservice-governance/gateway.md) - API gateway for request routing + +### Computation Governance Services +These services handle the core computation task lifecycle: +- [Entrance Service](./computation-governance/entrance.md) - Task submission and entrance point +- [Manager Service](./computation-governance/manager.md) - Resource and application management +- [ECM Service](./computation-governance/ecm.md) - Engine Connection Manager +- [JobHistory Service](./computation-governance/jobhistory.md) - Task execution history tracking + +### Public Enhancement Services +These services provide shared capabilities used across the platform: +- [Public Service](./public-enhancements/publicservice.md) - Core public services +- [Configuration Service](./public-enhancements/configuration.md) - Configuration management +- [BML Service](./public-enhancements/bml.md) - Big Data Material Library +- [DataSource Service](./public-enhancements/datasource.md) - Data source management +- [Context Service](./public-enhancements/context.md) - Context and variable sharing +- [Monitor Service](./microservice-governance/monitor.md) - System monitoring + +## RPC Development Example + +Linkis uses RPC (Remote Procedure Call) for inter-service communication. Here's a basic example of how to use RPC in Linkis: + +```java +// Get sender for target service +Sender sender = Sender.getSender("linkis-publicservice"); + +// Create request protocol +InsLabelAttachRequest request = new InsLabelAttachRequest(); +request.setServiceInstance(serviceInstance); +request.setLabels(labels); + +// Send request via RPC +sender.ask(request); +``` + +## Documentation Structure + +Each service module documentation includes: +- API interfaces +- Database table structures +- RPC methods provided to other services \ No newline at end of file diff --git a/docs/modules/computation-governance/README.md b/docs/modules/computation-governance/README.md new file mode 100644 index 0000000000..1f39355e66 --- /dev/null +++ b/docs/modules/computation-governance/README.md @@ -0,0 +1,157 @@ +# Computation Governance Services + +The computation governance services handle the core computation task lifecycle management in Linkis. + +## Service Modules + +- [Entrance Service](./entrance.md) - Task submission and entrance point +- [Manager Service](./manager.md) - Resource and application management +- [ECM Service](./ecm.md) - Engine Connection Manager +- [JobHistory Service](./jobhistory.md) - Task execution history tracking + +## Overview + +These services form the core of Linkis' computation governance capabilities, managing the complete lifecycle of computation tasks from submission to execution and monitoring. + +## Common Features + +### Task Lifecycle Management +- Task submission and validation +- Task scheduling and resource allocation +- Task execution monitoring +- Task result management +- Task error handling and recovery + +### Engine Management +- Dynamic engine connection creation +- Engine lifecycle management +- Engine resource monitoring +- Engine scaling capabilities + +### Resource Governance +- Multi-tenant resource isolation +- Load balancing across engines +- Resource usage tracking +- Quota management + +## API Interface Summary + +### Entrance Service APIs +- Task submission: `POST /api/entrance/submit` +- Task status query: `GET /api/entrance/{id}/status` +- Task progress: `GET /api/entrance/{id}/progress` +- Task log retrieval: `GET /api/entrance/{id}/log` +- Task cancellation: `GET /api/entrance/{id}/kill` + +### Manager Service APIs +- Engine instance management +- Resource allocation and monitoring +- Node status querying +- Engine creation requests + +### ECM Service APIs +- Engine connection management +- Engine lifecycle operations +- Resource reporting +- Engine metrics collection + +### JobHistory Service APIs +- Job history querying +- Job detail retrieval +- Job statistics reporting + +## Database Schema Summary + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +### Common Lock Table +```sql +CREATE TABLE `linkis_ps_common_lock` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `lock_object` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `locker` VARCHAR(255) CHARSET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'locker', + `time_out` longtext COLLATE utf8_bin, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_lock_object` (`lock_object`) +); +``` + +## RPC Methods Summary + +### Entrance Service RPCs +- `submitTask(TaskRequest request)` +- `getTaskStatus(String taskId)` +- `cancelTask(String taskId)` +- `getTaskResult(String taskId)` + +### Manager Service RPCs +- `requestEngine(EngineRequest request)` +- `releaseEngine(String engineId)` +- `getEngineStatus(String engineId)` +- `getNodeMetrics(String nodeId)` + +### ECM Service RPCs +- `createEngineConnection(EngineCreateRequest request)` +- `terminateEngineConnection(String engineId)` +- `reportEngineResourceUsage(String engineId, ResourceUsage usage)` +- `getEngineMetrics(String engineId)` + +### JobHistory Service RPCs +- `saveJobHistory(JobHistory history)` +- `queryJobHistory(JobHistoryQuery query)` +- `getJobDetails(Long jobId)` +- `updateJobStatus(Long jobId, String status)` + +## Dependencies + +- linkis-commons - Shared utilities +- linkis-protocol - Communication protocols +- linkis-rpc - Remote procedure calls +- Various engine connection plugins +- Spring Cloud ecosystem \ No newline at end of file diff --git a/docs/modules/computation-governance/ecm.md b/docs/modules/computation-governance/ecm.md new file mode 100644 index 0000000000..134c981783 --- /dev/null +++ b/docs/modules/computation-governance/ecm.md @@ -0,0 +1,635 @@ +# ECM Service + +The ECM (Engine Connection Manager) service manages the lifecycle of engine connections in the Linkis system. + +## Overview + +This service is responsible for managing the lifecycle of engine connections, including creating, starting, stopping, and monitoring engine instances. + +## Key Components + +### Core Classes +- `LinkisECMApplication` - Main application class +- Engine connection lifecycle management +- Engine resource monitoring +- Engine health checking + +### Features +- Engine connection creation and initialization +- Engine lifecycle management +- Resource allocation for engines +- Engine monitoring and health checking +- Engine termination and cleanup + +## API Interfaces + +### Download Engine Log +``` +GET /api/rest_j/v1/engineconnManager/downloadEngineLog +``` + +Parameters: +- `emInstance`: ECM instance (required) +- `instance`: Engine instance (required) +- `logDirSuffix`: Log directory suffix (required) +- `logType`: Log type (required) - stdout, stderr, gc, or yarnApp + +Response: +``` +Binary file download (log file content) +``` + +Error Codes: +- 11110: Log directory {0} does not exists.(日志目录 {0} 不存在.) +- 911115: failed to downLoad(下载失败) +- 911116: Download file has exceeded 100MB(下载文件已超过100M) +- 911117: Parameter {0} cannot be empty (参数 {0} 不能为空) +- 911118: logType only supports stdout, stderr, gc, yarnApp(logType仅支持stdout,stderr,gc,yarnApp) +- 911119: You {0} have no permission to download Log in ECM {1}(用户 {0} 无权限下载 ECM {1} 日志) + +Notes: +- Only supports GET method due to gateway forwarding rules +- File size limit is 100MB +- Supported log types: stdout, stderr, gc, yarnApp +- Requires user authentication and authorization checks +- Filename format in response: {instance}_{logType}.txt + +### List All ECMs +``` +GET /api/rest_j/v1/linkisManager/listAllEMs +``` + +Parameters: +- `instance`: ECM instance filter (optional) +- `nodeHealthy`: Node healthy status filter (optional) +- `owner`: Owner filter (optional) +- `tenantLabel`: Tenant label filter (optional) + +Response: +```json +{ + "method": "/api/linkisManager/listAllEMs", + "status": 0, + "message": "OK", + "data": { + "EMs": [ + { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "gz.bdz.bdplxxxxx.apache:9102", + "nodeHealthy": "Healthy", + "labels": [ + { + "stringValue": "gz.bdz.bdplxxxxx.apache:9102", + "labelKey": "emInstance" + } + ], + "owner": "hadoop", + "nodeStatus": "Healthy" + } + ] + } +} +``` + +Error Codes: +- 210003: Only admin can modify ECMs(只有管理员才能修改ECM) + +Notes: +- Requires admin privileges +- Returns list of all ECM instances with their status and labels + +### List All ECM Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllECMHealthyStatus +``` + +Parameters: +- `onlyEditable`: Boolean flag to return only editable statuses (optional) + +Response: +```json +{ + "method": "/api/linkisManager/listAllECMHealthyStatus", + "status": 0, + "message": "OK", + "data": { + "nodeHealthy": [ + "Healthy", + "UnHealthy", + "WARN", + "StockAvailable", + "StockUnavailable" + ] + } +} +``` + +Notes: +- Returns all possible ECM healthy status values +- When `onlyEditable` is true, returns only the statuses that can be modified + +### Modify ECM Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEMInfo +``` + +Parameters: +- `applicationName`: Application name (optional) +- `emStatus`: ECM status (optional) +- `instance`: ECM instance (required) +- `labels`: Labels list (optional) +- `labelKey`: Label key (optional) +- `description`: Description (optional) +- `stringValue`: String value (optional) + +Response: +```json +{ + "method": "/api/linkisManager/modifyEMInfo", + "status": 0, + "message": "success" +} +``` + +Error Codes: +- 210003: Failed to update label, include repeat labels(更新label失败，包含重复label) + +Notes: +- Allows modification of ECM instance information +- Supports updating labels, status, and description + +### Execute ECM Operation +``` +POST /api/rest_j/v1/linkisManager/executeECMOperation +``` + +Request Body: +```json +{ + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "gz.bdz.bdplxxxxx.apache:9102" + }, + "parameters": { + // Operation specific parameters + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperation", + "status": 0, + "message": "OK", + "data": { + // Operation result data + } +} +``` + +Error Codes: +- Various operation-specific error codes + +Notes: +- Executes administrative operations on ECM instances +- Requires appropriate permissions +- Operation parameters vary based on the specific operation being performed + +### Execute ECM Operation by Engine Connection +``` +POST /api/rest_j/v1/linkisManager/executeECMOperationByEC +``` + +Request Body: +```json +{ + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "gz.bdz.bdplxxxxx.apache:12295" + }, + "parameters": { + // Operation specific parameters + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperationByEC", + "status": 0, + "message": "OK", + "data": { + // Operation result data + } +} +``` + +Error Codes: +- Permission-related errors when user doesn't own the engine connection + +Notes: +- Executes ECM operations triggered by engine connections +- Validates that the user owns the engine connection or is an admin +- Operation parameters vary based on the specific operation being performed + +### Reset Resource +``` +GET /api/rest_j/v1/linkisManager/reset-resource +``` + +Parameters: +- `serviceInstance`: ECM service instance (optional) +- `username`: Username (optional) + +Response: +```json +{ + "method": "/api/linkisManager/reset-resource", + "status": 0, + "message": "OK", + "data": {} +} +``` + +Error Codes: +- Permission error when user is not admin + +Notes: +- Resets resource allocation for ECM instances or users +- Requires admin privileges +- Can reset resources for a specific ECM instance or user + +### Open Engine Log +``` +POST /api/rest_j/v1/linkisManager/openEngineLog +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "emInstance": "bdp110:9100", + "instance": "bdp110:21976", + "parameters": { + "logType": "stdout", + "fromLine": "0", + "pageSize": "1000" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/openEngineLog", + "status": 0, + "message": "OK", + "data": { + // Log content or operation result + } +} +``` + +Error Codes: +- Parameter validation errors +- Permission errors + +Notes: +- Opens and retrieves engine log content +- Supports different log types (stdout, stderr, gc, udfLog, yarnApp) +- Requires appropriate permissions + +### Task Prediction +``` +GET /api/rest_j/v1/linkisManager/task-prediction +``` + +Parameters: +- `username`: Username (optional) +- `engineType`: Engine type (required) +- `creator`: Creator (required) +- `clustername`: Cluster name (optional) +- `queueName`: Queue name (optional) +- `tenant`: Tenant (optional) + +Response: +```json +{ + "method": "/api/linkisManager/task-prediction", + "status": 0, + "message": "OK", + "data": { + "tenant": "tenant", + "userResource": {}, + "ecmResource": {}, + "yarnResource": {}, + "checkResult": true + } +} +``` + +Error Codes: +- Parameter validation errors + +Notes: +- Predicts if a task can be executed based on available resources +- Requires engineType and creator parameters +- Returns resource availability information + +### Get Engine Connection Info +``` +GET /api/rest_j/v1/linkisManager/ecinfo/get +``` + +Parameters: +- `ticketid`: Ticket ID (required) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/get", + "status": 0, + "message": "OK", + "data": { + "ecResourceInfoRecord": { + // Engine connection resource information + } + } +} +``` + +Error Codes: +- Ticket ID not found + +Notes: +- Retrieves engine connection information by ticket ID +- Requires user to be owner or admin + +### Delete Engine Connection Info +``` +DELETE /api/rest_j/v1/linkisManager/ecinfo/delete/{ticketid} +``` + +Parameters: +- `ticketid`: Ticket ID (required) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/delete/{ticketid}", + "status": 0, + "message": "OK", + "data": { + "ecResourceInfoRecord": { + // Deleted engine connection resource information + } + } +} +``` + +Error Codes: +- Ticket ID not found +- Permission errors + +Notes: +- Deletes engine connection information by ticket ID +- Requires user to be owner or admin + +### Query Engine Connection Resource History List +``` +GET /api/rest_j/v1/linkisManager/ecinfo/ecrHistoryList +``` + +Parameters: +- `instance`: Instance (optional) +- `creator`: Creator (optional) +- `startDate`: Start date (optional) +- `endDate`: End date (optional) +- `engineType`: Engine type (optional) +- `status`: Status (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecrHistoryList", + "status": 0, + "message": "OK", + "data": { + "engineList": [ + // Engine connection resource history records + ], + "totalPage": 100 + } +} +``` + +Error Codes: +- Parameter validation errors + +Notes: +- Queries engine connection resource history +- Supports filtering by various parameters +- Returns paginated results + +### Query Engine Connection List +``` +POST /api/rest_j/v1/linkisManager/ecinfo/ecList +``` + +Request Body: +```json +{ + "creators": ["IDE"], + "engineTypes": ["spark"], + "statuss": ["Running"], + "queueName": "default", + "ecInstances": ["instance1", "instance2"], + "crossCluster": false +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecList", + "status": 0, + "message": "OK", + "data": { + "ecList": [ + // Engine connection records + ] + } +} +``` + +Error Codes: +- Parameter validation errors +- Permission errors + +Notes: +- Queries engine connection list +- Requires admin privileges +- Supports filtering by various parameters + +## Database Table Structures + +The ECM service uses the following database tables for engine management: + +### Engine Connection Plugin BML Resources Table +```sql +CREATE TABLE `linkis_cg_engine_conn_plugin_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `engine_conn_type` varchar(100) NOT NULL COMMENT 'Engine type', + `version` varchar(100) COMMENT 'version', + `file_name` varchar(255) COMMENT 'file name', + `file_size` bigint(20) DEFAULT 0 NOT NULL COMMENT 'file size', + `last_modified` bigint(20) COMMENT 'File update time', + `bml_resource_id` varchar(100) NOT NULL COMMENT 'Owning system', + `bml_resource_version` varchar(200) NOT NULL COMMENT 'Resource owner', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'created time', + `last_update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'updated time', + PRIMARY KEY (`id`) +); +``` + +### Manager Engine EM Table +```sql +CREATE TABLE `linkis_cg_manager_engine_em` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `engine_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `em_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +); +``` + +### EC Resource Info Record Table +```sql +CREATE TABLE `linkis_cg_ec_resource_info_record` ( + `id` INT(20) NOT NULL AUTO_INCREMENT, + `label_value` VARCHAR(255) NOT NULL COMMENT 'ec labels stringValue', + `create_user` VARCHAR(128) NOT NULL COMMENT 'ec create user', + `service_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ec instance info', + `ecm_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ecm instance info ', + `ticket_id` VARCHAR(100) NOT NULL COMMENT 'ec ticket id', + `status` varchar(50) DEFAULT NULL COMMENT 'EC status: Starting,Unlock,Locked,Idle,Busy,Running,ShuttingDown,Failed,Success', + `log_dir_suffix` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'log path', + `request_times` INT(8) COMMENT 'resource request times', + `request_resource` VARCHAR(1020) COMMENT 'request resource', + `used_times` INT(8) COMMENT 'resource used times', + `used_resource` VARCHAR(1020) COMMENT 'used resource', + `metrics` TEXT DEFAULT NULL COMMENT 'ec metrics', + `release_times` INT(8) COMMENT 'resource released times', + `released_resource` VARCHAR(1020) COMMENT 'released resource', + `release_time` datetime DEFAULT NULL COMMENT 'released time', + `used_time` datetime DEFAULT NULL COMMENT 'used time', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + PRIMARY KEY (`id`), + KEY `idx_ticket_id` (`ticket_id`), + UNIQUE KEY `uniq_tid_lv` (`ticket_id`,`label_value`), + UNIQUE KEY `uniq_sinstance_status_cuser_ctime` (`service_instance`, `status`, `create_user`, `create_time`) +); +``` + +## RPC Methods + +The ECM service provides several RPC methods for engine management: + +### Engine Management RPCs + +#### createEngineConnection +Creates a new engine connection: +```java +EngineConnection createEngineConnection(EngineCreateRequest request) +``` + +#### executeCode +Executes code on an engine: +```java +ExecutionResult executeCode(String engineId, String code, String runType) +``` + +#### getEngineStatus +Retrieves the status of an engine: +```java +EngineStatus getEngineStatus(String engineId) +``` + +#### terminateEngine +Terminates an engine connection: +```java +void terminateEngine(String engineId) +``` + +#### listEngines +Lists all engine connections: +```java +List listEngines() +``` + +#### getEngineMetrics +Retrieves metrics for an engine: +```java +EngineMetrics getEngineMetrics(String engineId) +``` + +### Resource Management RPCs + +#### getResourceUsage +Retrieves resource usage for an engine: +```java +ResourceUsage getResourceUsage(String engineId) +``` + +#### updateResource +Updates resource allocation for an engine: +```java +void updateResource(String engineId, ResourceRequest request) +``` + +#### reportResourceUsage +Reports resource usage from an engine: +```java +void reportResourceUsage(String engineId, ResourceUsage usage) +``` + +### Engine Communication RPCs + +#### sendEngineCommand +Sends a command to an engine: +```java +CommandResponse sendEngineCommand(String engineId, EngineCommand command) +``` + +#### getEngineLogs +Retrieves logs from an engine: +```java +EngineLogs getEngineLogs(String engineId, int fromLine, int lines) +``` + +## Dependencies + +- linkis-engineconn-manager-core +- linkis-engineconn-plugin-core +- linkis-rpc +- linkis-protocol +- linkis-manager-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ECMRestfulApi: `linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-server/src/main/java/org/apache/linkis/ecm/restful/ECMRestfulApi.java` +- EMRestfulApi: `linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/am/restful/EMRestfulApi.java` +- ECResourceInfoRestfulApi: `linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/am/restful/ECResourceInfoRestfulApi.java` + +### MyBatis XML Files +The ECM service primarily uses the Manager service's persistence layer, which includes: +- LabelManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/LabelManagerMapper.xml` +- ResourceManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/ResourceManagerMapper.xml` +- NodeManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml` +- NodeMetricManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeMetricManagerMapper.xml` \ No newline at end of file diff --git a/docs/modules/computation-governance/entrance.md b/docs/modules/computation-governance/entrance.md new file mode 100644 index 0000000000..6071b33f9a --- /dev/null +++ b/docs/modules/computation-governance/entrance.md @@ -0,0 +1,726 @@ +# Entrance Service + +The Entrance service serves as the entry point for computation task submissions in the Linkis system. + +## Overview + +This service is responsible for receiving user computation requests, parsing them, validating them, and coordinating their execution through the appropriate engine connections. It acts as the primary interface between users and the computation execution layer. + +## Key Components + +### Core Classes +- `LinkisEntranceApplication` - Main application class +- Task submission handling +- Task parsing and validation +- Task scheduling coordination +- Task execution monitoring +- Task result management + +### Features +- Task submission and management +- Code parsing and validation +- Engine routing and allocation +- Result set management +- Log retrieval and management + +## API Interfaces + +### Task Execution +``` +POST /api/entrance/execute +``` + +Parameters (in request body): +- `executionContent`: Contains the code to execute and run type + - `code`: The actual code to execute + - `runType`: Type of execution (sql, python, scala, etc.) +- `params`: Parameters for execution + - `variable`: Variables for the execution + - `configuration`: Configuration parameters (runtime, special) +- `source`: Source information + - `scriptPath`: Path to the script file +- `labels`: Labels for engine selection + - `engineType`: Type and version of engine (spark-2.4.3, hive-2.1.1, etc.) + - `userCreator`: User and creator information + +Response: +```json +{ + "method": "/api/entrance/execute", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If parsing or execution fails, the error will be stored in the job request and returned in the response +- Permission errors if user is not authorized to execute + +Notes: +- Returns both taskID (database ID) and execID (execution ID) +- The execID is used for subsequent operations on the task +- User authentication is required + +### Task Submission +``` +POST /api/entrance/submit +``` + +Parameters (in request body): +- Same as execute API + +Response: +```json +{ + "method": "/api/entrance/submit", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If parsing or execution fails, the error will be stored in the job request and returned in the response +- Permission errors if user is not authorized to submit + +Notes: +- Functionally similar to execute but with different endpoint +- Returns both taskID (database ID) and execID (execution ID) +- User authentication is required + +### Task Status Query +``` +GET /api/entrance/{id}/status +``` + +Parameters: +- `id`: The execution ID or task ID +- `taskID` (optional): The ID of the task to query + +Response: +```json +{ + "method": "/api/entrance/{id}/status", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "status": "Running", + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If there's an exception during status retrieval, error is returned + +Notes: +- Supports both execID and taskID as the path parameter +- Status values include: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout +- For completed jobs, status is retrieved from job history + +### Task Progress +``` +GET /api/entrance/{id}/progress +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/progress", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "progress": "0.75", + "execID": "exec-id-12345", + "progressInfo": [ + { + "id": "stage1", + "succeedTasks": 5, + "failedTasks": 0, + "runningTasks": 2, + "totalTasks": 10 + } + ] + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If progress information is not yet available, error is returned + +Notes: +- Progress is a value between 0 and 1 +- ProgressInfo provides detailed information about execution stages +- For completed jobs, returns 1.0 progress + +### Task Progress with Resource Info +``` +GET /api/entrance/{id}/progressWithResource +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/progressWithResource", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "progress": "0.75", + "execID": "exec-id-12345", + "progressInfo": [ + { + "id": "stage1", + "succeedTasks": 5, + "failedTasks": 0, + "runningTasks": 2, + "totalTasks": 10 + } + ], + "jobYarnMetrics": { + "jobYarnResource": [ + { + "applicationId": "application_1234567890123_0001", + "queueCores": 2, + "queueMemory": 4096, + "usedCores": 1, + "usedMemory": 2048, + "resourceType": "YARN" + } + ] + } + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If progress information is not yet available, error is returned + +Notes: +- Includes YARN resource metrics in addition to progress information +- Provides detailed resource usage information for YARN-based engines + +### Task Log Retrieval +``` +GET /api/entrance/{id}/log +``` + +Parameters: +- `id`: The execution ID +- `fromLine` (optional): Starting line number (default: 0) +- `size` (optional): Number of lines to retrieve (default: 100) +- `distinctLevel` (optional): Whether to separate logs by level (default: true) + +Response: +```json +{ + "method": "/api/entrance/{id}/log", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "log": ["log line 1", "log line 2", "log line 3"], + "fromLine": 1, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job has completed, suggests downloading log file instead +- If log cannot be retrieved, returns appropriate error + +Notes: +- For distinctLevel=true, returns array with 4 elements (different log levels) +- For distinctLevel=false, returns concatenated string of logs +- Size parameter has a maximum limit (10000) + +### Task Cancellation +``` +GET /api/entrance/{id}/kill +``` + +Parameters: +- `id`: The execution ID +- `taskID` (optional): The ID of the task to cancel + +Response: +```json +{ + "method": "/api/entrance/{id}/kill", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job is already completed, returns error that kill is not supported +- If user doesn't have permission to kill the job, returns permission error +- If exception occurs during kill, returns error with exception details + +Notes: +- Updates job status to Cancelled in database +- For jobs not found in memory, performs force kill using job history + +### Batch Task Cancellation +``` +POST /api/entrance/{id}/killJobs +``` + +Request Body: +```json +{ + "idList": ["exec-id-1", "exec-id-2"], + "taskIDList": [12345, 12346] +} +``` + +Parameters: +- `id`: The strong execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/killJobs", + "status": 0, + "message": "success", + "data": { + "messages": [ + { + "method": "/api/entrance/exec-id-1/kill", + "status": 0, + "message": "Successfully killed the job(成功kill了job)" + }, + { + "method": "/api/entrance/exec-id-2/kill", + "status": 0, + "message": "Successfully killed the job(成功kill了job)" + } + ] + } +} +``` + +Error Cases: +- If idList and taskIDList have different lengths, returns error +- If parameters are not arrays, returns error +- Individual job kill errors are returned in the messages array + +Notes: +- Processes each job in the lists and returns individual results +- For jobs not found in memory, performs force kill using job history + +### Task Pause +``` +GET /api/entrance/{id}/pause +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/pause", + "status": 0, + "message": "success to pause job (成功pause了job)", + "data": { + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job cannot be found, returns appropriate error +- If exception occurs during pause, returns error + +Notes: +- Pause functionality implementation may be incomplete (TODO in code) + +### Update Route Label +``` +POST /api/entrance/operation/label/update +``` + +Request Body: +```json +{ + "routeLabel": "new-route-label" +} +``` + +Parameters: +- Requires admin privileges + +Response: +```json +{ + "method": "/api/entrance/operation/label/update", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Updates the route label for the entrance instance +- Used for routing purposes in distributed environments + +### Mark Offline +``` +GET /api/entrance/operation/label/markoffline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/markoffline", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Marks the entrance instance as offline +- Updates all non-execution task instances + +### Back Online +``` +GET /api/entrance/operation/label/backonline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/backonline", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Removes the offline label from the entrance instance + +### Check Online Status +``` +GET /api/entrance/operation/label/isOnline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/isOnline", + "status": 0, + "message": "success", + "data": { + "isOnline": true + } +} +``` + +Notes: +- Checks if the entrance instance is currently online + +### Get Task Info +``` +GET /api/entrance/operation/metrics/taskinfo +``` + +Parameters: +- `user` (optional): Filter by user +- `creator` (optional): Filter by creator +- `ecType` (optional): Filter by engine type + +Response: +```json +{ + "method": "/api/entrance/operation/metrics/taskinfo", + "status": 0, + "message": "success", + "data": { + "taskNumber": 5, + "runningNumber": 2, + "queuedNumber": 3 + } +} +``` + +Error Cases: +- Non-admin users cannot view other users' task information + +Notes: +- For admin users, can view any user's task information +- For non-admin users, can only view their own task information +- Returns counts of total, running, and queued tasks + +### Get Running Task Count +``` +GET /api/entrance/operation/metrics/runningtask +``` + +Response: +```json +{ + "method": "/api/entrance/operation/metrics/runningtask", + "status": 0, + "message": "success", + "data": { + "runningTaskNumber": 5, + "isCompleted": false + } +} +``` + +Notes: +- Returns the number of currently running tasks +- isCompleted indicates if there are no running tasks + +### Kill Consumer +``` +GET /api/entrance/operation/consumer/kill +``` + +Parameters: +- `groupName`: Name of the consumer group to kill + +Response: +```json +{ + "method": "/api/entrance/operation/consumer/kill", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Destroys the specified consumer group +- Requires admin privileges + +### Get Consumer Info +``` +GET /api/entrance/operation/consumer/info +``` + +Response: +```json +{ + "method": "/api/entrance/operation/consumer/info", + "status": 0, + "message": "success", + "data": { + "consumerNum": 3 + } +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Returns the number of consumer groups +- Requires admin privileges + +## Database Table Structures + +The Entrance service uses the following database tables from the job history system: + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The Entrance service provides several RPC methods for inter-service communication: + +### Task Management RPCs + +#### submitTask +Submits a task for execution: +```java +JobRespProtocol submitTask(JobReqInsert request) +``` + +#### updateTask +Updates a task: +```java +JobRespProtocol updateTask(JobReqUpdate request) +``` + +#### batchUpdateTasks +Batch updates tasks: +```java +JobRespProtocol batchUpdateTasks(JobReqBatchUpdate request) +``` + +#### queryTask +Queries a task: +```java +JobRespProtocol queryTask(JobReqQuery request) +``` + +#### readAllTasks +Reads all tasks: +```java +JobRespProtocol readAllTasks(JobReqReadAll request) +``` + +#### getTaskStatus +Retrieves the status of a task: +```java +String getTaskStatus(String taskId) +``` + +#### cancelTask +Cancels a running task: +```java +void cancelTask(String taskId) +``` + +#### getTaskResult +Retrieves the result of a completed task: +```java +TaskResult getTaskResult(String taskId) +``` + +#### getTaskProgress +Retrieves the progress of a task: +```java +TaskProgress getTaskProgress(String taskId) +``` + +### Engine Management RPCs + +#### requestEngine +Requests an engine for task execution: +```java +EngineConnection requestEngine(EngineRequest request) +``` + +#### releaseEngine +Releases an engine after task completion: +```java +void releaseEngine(String engineId) +``` + +#### getEngineStatus +Retrieves the status of an engine: +```java +EngineStatus getEngineStatus(String engineId) +``` + +### Log Management RPCs + +#### getTaskLog +Retrieves logs for a specific task: +```java +TaskLog getTaskLog(String taskId, int fromLine, int pageSize) +``` + +#### appendTaskLog +Appends log entries for a task: +```java +void appendTaskLog(String taskId, List logLines) +``` + +## Dependencies + +- linkis-scheduler +- linkis-protocol +- linkis-rpc +- linkis-storage +- linkis-computation-governance-common +- linkis-computation-orchestrator +- linkis-pes-client +- linkis-io-file-client +- linkis-pes-rpc-client +- linkis-ps-common-lock + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- EntranceRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java` +- EntranceLabelRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java` +- EntranceMetricRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java` +- EntranceConsumerRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceConsumerRestfulApi.java` + +### MyBatis XML Files +The Entrance service uses the JobHistory service's persistence layer, which includes: +- JobHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/mysql/JobHistoryMapper.xml` +- JobDetailMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDetailMapper.xml` +- JobStatisticsMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobStatisticsMapper.xml` +- JobDiagnosisMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml` \ No newline at end of file diff --git a/docs/modules/computation-governance/jobhistory.md b/docs/modules/computation-governance/jobhistory.md new file mode 100644 index 0000000000..7e875ce2f6 --- /dev/null +++ b/docs/modules/computation-governance/jobhistory.md @@ -0,0 +1,481 @@ +# JobHistory Service + +The JobHistory service tracks and manages the execution history of tasks in the Linkis system. + +## Overview + +This service provides task execution history tracking, including task status, execution time, results, and error information. + +## Key Components + +### Core Classes +- `LinkisJobHistoryApp` - Main application class +- Task history storage and retrieval +- Task statistics and analytics +- Task search and filtering + +### Features +- Task execution history tracking +- Task result storage and retrieval +- Task performance metrics +- Task search and filtering capabilities +- Task statistics and reporting + +## API Interfaces + +### Get Task By ID +``` +GET /api/rest_j/v1/jobhistory/{id}/get +``` + +Parameters: +- `id`: Job ID (required) +- `brief`: Whether to return brief info only (optional) + +Response: +```json +{ + "method": "/api/jobhistory/{id}/get", + "status": 0, + "message": "success", + "data": { + "task": { + "jobId": "12345", + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "executionCode": "SELECT * FROM table", + "resultLocation": "/path/to/result", + "errorCode": null, + "errorDesc": null, + "progress": "1.0", + "costTime": 300000 + } + } +} +``` + +### List Job History +``` +GET /api/rest_j/v1/jobhistory/list +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `status`: Task status to filter by (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) +- `taskID`: Task ID to filter by (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) +- `isAdminView`: Whether to view as admin (optional) +- `isDeptView`: Whether to view as department admin (optional) +- `instance`: Instance to filter by (optional) +- `engineInstance`: Engine instance to filter by (optional) +- `runType`: Run type to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/list", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "executionCode": "SELECT * FROM table", + "resultLocation": "/path/to/result", + "errorCode": null, + "errorDesc": null, + "progress": "1.0", + "costTime": 300000 + } + ], + "totalPage": 1 + } +} +``` + +### List Undone Tasks +``` +GET /api/rest_j/v1/jobhistory/listundonetasks +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `status`: Task status to filter by (optional, default: "Running,Inited,Scheduled") +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) +- `startTaskID`: Start task ID (optional) +- `engineType`: Engine type to filter by (optional) +- `creator`: Creator to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/listundonetasks", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Running", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00" + } + ], + "totalPage": 1 + } +} +``` + +### List By Task IDs +``` +GET /api/rest_j/v1/jobhistory/list-taskids +``` + +Parameters: +- `taskID`: Comma-separated list of task IDs (required) + +Response: +```json +{ + "method": "/api/jobhistory/list-taskids", + "status": 0, + "message": "success", + "data": { + "jobHistoryList": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00" + } + ] + } +} +``` + +### Job Extra Info +``` +GET /api/rest_j/v1/jobhistory/job-extra-info +``` + +Parameters: +- `jobId`: Job ID (required) + +Response: +```json +{ + "method": "/api/jobhistory/job-extra-info", + "status": 0, + "message": "success", + "data": { + "metricsMap": { + "executionCode": "SELECT * FROM table", + "runtime": "300000" + } + } +} +``` + +### List Duration Top +``` +GET /api/rest_j/v1/jobhistory/listDurationTop +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) + +Response: +```json +{ + "method": "/api/jobhistory/listDurationTop", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "costTime": 300000 + } + ] + } +} +``` + +### Task Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/taskCount +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/taskCount", + "status": 0, + "message": "success", + "data": { + "sumCount": 100, + "succeedCount": 95, + "failedCount": 5, + "cancelledCount": 0 + } +} +``` + +### Engine Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/engineCount +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/engineCount", + "status": 0, + "message": "success", + "data": { + "countEngine": 100, + "countEngineSucceed": 95, + "countEngineFailed": 5, + "countEngineShutting": 0 + } +} +``` + +### Add Observe Info +``` +POST /api/rest_j/v1/jobhistory/setting/addObserveInfo +``` + +Request Body: +```json +{ + "taskId": 12345, + "receiver": "testuser", + "extra": { + "title": "Task Alert", + "detail": "Task execution alert" + }, + "monitorLevel": "HIGH", + "subSystemId": "1" +} +``` + +Response: +```json +{ + "method": "/api/jobhistory/setting/addObserveInfo", + "status": 0, + "message": "success" +} +``` + +### Delete Observe Info +``` +GET /api/rest_j/v1/jobhistory/setting/deleteObserveInfo +``` + +Parameters: +- `taskId`: Task ID (required) + +Response: +```json +{ + "method": "/api/jobhistory/setting/deleteObserveInfo", + "status": 0, + "message": "success" +} +``` + +## Database Table Structures + +The JobHistory service uses the following database tables from the linkis_ddl.sql file: + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The JobHistory service provides several RPC methods for job history management: + +### Job History RPCs + +#### recordJob +Records a job execution: +```java +void recordJob(JobRecordRequest request) +``` + +#### updateJobStatus +Updates the status of a job: +```java +void updateJobStatus(String jobId, JobStatus status) +``` + +#### getJobHistory +Retrieves job history: +```java +JobHistory getJobHistory(String jobId) +``` + +#### searchJobs +Searches for jobs based on criteria: +```java +List searchJobs(JobSearchCriteria criteria) +``` + +#### getJobDetails +Retrieves detailed job information: +```java +JobDetails getJobDetails(Long jobId) +``` + +#### deleteJobHistory +Deletes job history records: +```java +void deleteJobHistory(List jobIds) +``` + +### Statistics RPCs + +#### recordStatistics +Records job statistics: +```java +void recordStatistics(JobStatistics statistics) +``` + +#### getStatistics +Retrieves job statistics: +```java +JobStatistics getStatistics(String jobId) +``` + +#### getStatisticsByUser +Retrieves job statistics for a user: +```java +List getStatisticsByUser(String username, Date startDate, Date endDate) +``` + +#### getStatisticsByEngine +Retrieves job statistics by engine type: +```java +List getStatisticsByEngine(String engineType, Date startDate, Date endDate) +``` + +## Dependencies + +- linkis-mybatis +- linkis-rpc +- linkis-protocol +- linkis-common +- linkis-computation-governance-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- QueryRestfulApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java` +- StatisticsRestfulApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/StatisticsRestfulApi.java` +- JobhistorySettingApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/JobhistorySettingApi.java` + +### MyBatis XML Files +- JobHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/mysql/JobHistoryMapper.xml` +- JobDetailMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDetailMapper.xml` +- JobStatisticsMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobStatisticsMapper.xml` +- JobDiagnosisMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml` +- JobAiHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobAiHistoryMapper.xml` diff --git a/docs/modules/computation-governance/manager.md b/docs/modules/computation-governance/manager.md new file mode 100644 index 0000000000..a74a77ed87 --- /dev/null +++ b/docs/modules/computation-governance/manager.md @@ -0,0 +1,1010 @@ +# Manager Service + +The Manager service provides resource and application management capabilities for the Linkis system. + +## Overview + +This service manages the resources and applications in the Linkis system, including node management, resource allocation, label management, and engine lifecycle management. + +## Key Components + +### Core Classes +- `LinkisManagerApplication` - Main application class +- Node management +- Resource management +- Label management +- Engine lifecycle management + +### Features +- Node registration and management +- Resource allocation and monitoring +- Label-based routing +- Engine instance management +- Load balancing + +## API Interfaces + +### ECM (EngineConnManager) Management + +#### List All ECMs +``` +GET /api/rest_j/v1/linkisManager/listAllEMs +``` + +Parameters: +- `instance` (optional): Filter by instance name +- `nodeHealthy` (optional): Filter by node health status (Healthy, UnHealthy, WARN, StockAvailable, StockUnavailable) +- `owner` (optional): Filter by owner +- `tenantLabel` (optional): Filter by tenant label + +Response: +```json +{ + "method": "/api/linkisManager/listAllEMs", + "status": 0, + "message": "success", + "data": { + "EMs": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + }, + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ], + "nodeHealthy": "Healthy", + "owner": "testuser" + } + ] + } +} +``` + +Error Cases: +- Only admin users can access this API +- If parameters are invalid, appropriate error messages are returned + +Notes: +- Requires admin privileges +- Results can be filtered and sorted by various criteria +- Returns EMNodeVo objects with detailed information about each ECM + +#### List All ECM Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllECMHealthyStatus +``` + +Parameters: +- `onlyEditable` (optional): If true, returns only editable statuses (Healthy, UnHealthy, WARN, StockAvailable, StockUnavailable) + +Response: +```json +{ + "method": "/api/linkisManager/listAllECMHealthyStatus", + "status": 0, + "message": "success", + "data": { + "nodeHealthy": ["Healthy", "UnHealthy", "WARN", "StockAvailable", "StockUnavailable"] + } +} +``` + +Notes: +- Returns all possible NodeHealthy enum values +- With onlyEditable=true, returns only the statuses that can be modified by users + +#### Modify ECM Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEMInfo +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102", + "emStatus": "Healthy", + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/modifyEMInfo", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- Only admin users can modify ECM info +- If applicationName or instance is null, returns error +- If labels contain duplicates, returns error +- If label values are invalid, returns error + +Notes: +- Requires admin privileges +- Can update both EM status and labels +- Supports UserModifiable labels with value validation + +#### Execute ECM Operation +``` +POST /api/rest_j/v1/linkisManager/executeECMOperation +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102", + "parameters": { + "operation": "stopEngine", + "engineConnInstance": "bdp110:12295" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperation", + "status": 0, + "message": "success", + "data": { + "result": "Operation executed successfully", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If ECM node doesn't exist, returns error +- If operation parameters are invalid, returns error + +Notes: +- Supports various admin operations (configurable via AMConfiguration.ECM_ADMIN_OPERATIONS) +- For log operations, automatically fills in logDirSuffix if not provided +- Validates user permissions for admin operations + +#### Execute ECM Operation By EC +``` +POST /api/rest_j/v1/linkisManager/executeECMOperationByEC +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "parameters": { + "operation": "stopEngine" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperationByEC", + "status": 0, + "message": "success", + "data": { + "result": "Operation executed successfully", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If engine node doesn't exist, returns error +- If operation parameters are invalid, returns error + +Notes: +- User must be owner of the engine or admin +- Delegates to executeECMOperation after validating permissions + +#### Open Engine Log +``` +POST /api/rest_j/v1/linkisManager/openEngineLog +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "emInstance": "bdp110:9100", + "instance": "bdp110:21976", + "parameters": { + "logType": "stdout", + "fromLine": "0", + "pageSize": "1000" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/openEngineLog", + "status": 0, + "message": "success", + "data": { + "result": "Log content...", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to access logs, returns error +- If log type is invalid, returns error +- If engine instance doesn't exist, returns error + +Notes: +- Supported log types: stdout, stderr, gc, udfLog, yarnApp +- Automatically fills in logDirSuffix if not provided +- Validates user permissions (must be owner or admin) + +#### Task Prediction +``` +GET /api/rest_j/v1/linkisManager/task-prediction +``` + +Parameters: +- `username` (optional): User name (defaults to current user) +- `engineType` (required): Engine type (spark/hive/etc.) +- `creator` (required): Creator application +- `clustername` (optional): Cluster name +- `queueName` (optional): Queue name +- `tenant` (optional): Tenant + +Response: +```json +{ + "method": "/api/linkisManager/task-prediction", + "status": 0, + "message": "success", + "data": { + "tenant": "tenant", + "userResource": {...}, + "ecmResource": {...}, + "yarnResource": {...}, + "checkResult": true + } +} +``` + +Error Cases: +- If engineType or creator is null, returns error +- If resource check fails, returns error + +Notes: +- Checks if user can create an engine for specified parameters +- Returns detailed resource information for user, ECM, and YARN + +#### Reset Resource +``` +GET /api/rest_j/v1/linkisManager/reset-resource +``` + +Parameters: +- `serviceInstance` (optional): Service instance to reset +- `username` (optional): User name to reset + +Response: +```json +{ + "method": "/api/linkisManager/reset-resource", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- Only admin users can reset resources + +Notes: +- Requires admin privileges +- Resets resource allocations for specified instance or user + +### Engine Management + +#### Ask Engine Connection +``` +POST /api/rest_j/v1/linkisManager/askEngineConn +``` + +Request Body: +```json +{ + "labels": { + "engineType": "spark-2.4.3", + "userCreator": "testuser-IDE" + }, + "timeOut": 30000, + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/askEngineConn", + "status": 0, + "message": "create engineConn ended.", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Starting", + "ticketId": "ticket-12345", + "ecmServiceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + } + } + } +} +``` + +Error Cases: +- If timeout is invalid, uses default timeout +- If engine creation fails, returns error with retry information + +Notes: +- First attempts to reuse existing engines +- If no suitable engine found, creates a new one +- Supports async engine creation with timeout handling + +#### Create Engine Connection +``` +POST /api/rest_j/v1/linkisManager/createEngineConn +``` + +Request Body: +```json +{ + "labels": { + "engineType": "spark-2.4.3", + "userCreator": "testuser-IDE" + }, + "timeout": 30000, + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/createEngineConn", + "status": 0, + "message": "create engineConn succeed.", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Starting", + "ticketId": "ticket-12345" + } + } +} +``` + +Error Cases: +- If timeout is invalid, uses default timeout +- If engine creation fails, returns error with retry information + +Notes: +- Always creates a new engine (doesn't attempt reuse) +- Supports timeout configuration +- Returns EngineNode information with service instance and ticket ID + +#### Get Engine Connection +``` +POST /api/rest_j/v1/linkisManager/getEngineConn +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/getEngineConn", + "status": 0, + "message": "success", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "ticketId": "ticket-12345" + } + } +} +``` + +Error Cases: +- If user doesn't have permission to access engine, returns error +- If engine instance doesn't exist, returns error + +Notes: +- User must be owner of the engine or admin +- Can retrieve engine info by service instance or ticket ID +- Returns EC metrics if available + +#### Kill Engine Connection +``` +POST /api/rest_j/v1/linkisManager/killEngineConn +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/killEngineConn", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If user doesn't have permission to kill engine, returns error +- If engine instance doesn't exist, returns error + +Notes: +- User must be owner of the engine or admin +- Sends EngineStopRequest to engine stop service +- Logs kill operation + +#### Kill ECM Engines +``` +POST /api/rest_j/v1/linkisManager/rm/killUnlockEngineByEM +``` + +Request Body: +```json +{ + "instance": "bdp110:9210" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/killUnlockEngineByEM", + "status": 0, + "message": "Kill engineConn succeed.", + "data": { + "result": {...} + } +} +``` + +Error Cases: +- Only admin users can kill engines by ECM +- If instance parameter is null, returns error + +Notes: +- Requires admin privileges +- Kills all unlocked engines under specified ECM +- Returns result information + +#### Kill Multiple Engines +``` +POST /api/rest_j/v1/linkisManager/rm/enginekill +``` + +Request Body: +```json +[ + { + "applicationName": "linkis-cg-engineconn", + "engineInstance": "bdp110:12295" + } +] +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/enginekill", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If engine instances don't exist, logs error but continues + +Notes: +- Kills multiple engines in a single request +- No permission check (uses internal sender) + +#### Kill Multiple Engines Async +``` +POST /api/rest_j/v1/linkisManager/rm/enginekillAsyn +``` + +Request Body: +```json +{ + "instances": ["bdp110:12295", "bdp110:12296"] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/enginekillAsyn", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If user is not admin and doesn't have valid token, returns error +- If instances parameter is null or empty, returns error +- If instances parameter parsing fails, returns error + +Notes: +- Requires admin privileges or valid admin token +- Asynchronously stops engines with metrics update +- Supports batch killing of multiple engine instances + +#### List User Engines +``` +GET /api/rest_j/v1/linkisManager/listUserEngines +``` + +Response: +```json +{ + "method": "/api/linkisManager/listUserEngines", + "status": 0, + "message": "success", + "data": { + "engines": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "owner": "testuser", + "engineType": "spark" + } + ] + } +} +``` + +Notes: +- Returns engines owned by the current user +- Lists all engine nodes for the user + +#### List ECM Engines +``` +POST /api/rest_j/v1/linkisManager/listEMEngines +``` + +Request Body: +```json +{ + "em": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + } + }, + "emInstance": "bdp110:9102" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/listEMEngines", + "status": 0, + "message": "success", + "data": { + "engines": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "owner": "testuser", + "engineType": "spark" + } + ] + } +} +``` + +Error Cases: +- Only admin users can list ECM engines +- If parameters are invalid, returns error + +Notes: +- Requires admin privileges +- Supports filtering by EM instance, node status, engine type, and owner +- Returns AMEngineNodeVo objects with detailed engine information + +#### Modify Engine Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEngineInfo +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ], + "nodeHealthy": "Healthy" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/modifyEngineInfo", + "status": 0, + "message": "success to update engine information(更新引擎信息成功)" +} +``` + +Error Cases: +- Only admin users can modify engine info +- If applicationName or instance is null, returns error +- If labels contain duplicates, returns error + +Notes: +- Requires admin privileges +- Can update both engine labels and health status +- Health status updates only support Healthy and UnHealthy values + +#### Batch Set Engine To UnHealthy +``` +POST /api/rest_j/v1/linkisManager/batchSetEngineToUnHealthy +``` + +Request Body: +```json +{ + "instances": [ + { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/batchSetEngineToUnHealthy", + "status": 0, + "message": "success to update engine information(批量更新引擎健康信息成功)" +} +``` + +Error Cases: +- Only admin users can set engine health status +- If instances parameter is null, returns error + +Notes: +- Requires admin privileges +- Sets multiple engines to UnHealthy status +- Logs batch update operation + +#### List All Node Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllNodeHealthyStatus +``` + +Parameters: +- `onlyEditable` (optional): If true, returns only editable statuses + +Response: +```json +{ + "method": "/api/linkisManager/listAllNodeHealthyStatus", + "status": 0, + "message": "success", + "data": { + "nodeStatus": ["Starting", "Unlock", "Locked", "Idle", "Busy", "Running", "ShuttingDown", "Failed", "Success"] + } +} +``` + +Notes: +- Returns all possible NodeStatus enum values +- With onlyEditable parameter, behavior is the same (returns all statuses) + +#### Execute Engine Conn Operation +``` +POST /api/rest_j/v1/linkisManager/executeEngineConnOperation +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "parameters": { + "operation": "someOperation" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeEngineConnOperation", + "status": 0, + "message": "success", + "data": { + "result": "Operation result...", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If engine instance doesn't exist, returns error +- If operation fails, returns error details + +Notes: +- User must be owner of the engine or admin +- Executes arbitrary operations on engine nodes +- Returns operation result and error information + +#### Kill Engines By Creator Or EngineType +``` +POST /api/rest_j/v1/linkisManager/rm/killEngineByCreatorEngineType +``` + +Request Body: +```json +{ + "creator": "IDE", + "engineType": "hive-2.3.3" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/killEngineByCreatorEngineType", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- Only admin users can kill engines by creator or engine type +- If creator or engineType parameters are null, returns error + +Notes: +- Requires admin privileges +- Kills all engines matching creator and engine type +- Supports cross-cluster killing with additional parameters + +### EC Resource Info Management + +#### Get EC Resource Info +``` +GET /api/rest_j/v1/linkisManager/ecinfo/get +``` + +Parameters: +- `ticketid`: Ticket ID + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/get", + "status": 0, + "message": "success", + "data": { + "ecResourceInfoRecord": { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running", + "usedResource": "{\"cpu\": 2, \"memory\": \"2G\"}", + "releasedResource": "{\"cpu\": 0, \"memory\": \"0G\"}", + "requestResource": "{\"cpu\": 2, \"memory\": \"2G\"}" + } + } +} +``` + +Error Cases: +- If ticket ID doesn't exist, returns error +- If user doesn't have permission to access resource info, returns error + +Notes: +- User must be creator of the resource or admin +- Returns detailed EC resource information record +- Includes resource usage statistics + +#### Delete EC Resource Info +``` +DELETE /api/rest_j/v1/linkisManager/ecinfo/delete/{ticketid} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/delete/ticket-12345", + "status": 0, + "message": "success", + "data": { + "ecResourceInfoRecord": { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running" + } + } +} +``` + +Error Cases: +- If ticket ID doesn't exist, returns error +- If user doesn't have permission to delete resource info, returns error + +Notes: +- User must be creator of the resource or admin +- Deletes EC resource information record from database +- Returns deleted record information + +#### Query EC Resource History List +``` +GET /api/rest_j/v1/linkisManager/ecinfo/ecrHistoryList +``` + +Parameters: +- `instance` (optional): Filter by instance +- `creator` (optional): Filter by creator +- `startDate` (optional): Filter by start date +- `endDate` (optional): Filter by end date (defaults to current date) +- `engineType` (optional): Filter by engine type +- `status` (optional): Filter by status +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecrHistoryList", + "status": 0, + "message": "success", + "data": { + "engineList": [ + { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running", + "usedResource": { + "cpu": 2, + "memory": "2G" + }, + "releasedResource": { + "cpu": 0, + "memory": "0G" + }, + "requestResource": { + "cpu": 2, + "memory": "2G" + } + } + ], + "totalPage": 1 + } +} +``` + +Error Cases: +- If creator parameter is invalid, returns error +- If date parameters are invalid, uses defaults + +Notes: +- Admin users can view all records, regular users only their own +- Supports date range filtering +- Supports pagination +- Converts resource strings to maps for easier consumption + +#### Query EC List +``` +POST /api/rest_j/v1/linkisManager/ecinfo/ecList +``` + +Request Body: +```json +{ + "creators": ["testuser"], + "engineTypes": ["spark-2.4.3"], + "statuss": ["Running"], + "queueName": "default", + "ecInstances": ["bdp110:12295"], + "crossCluster": false +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecList", + "status": 0, + "message": "success", + "data": { + "ecList": [ + { + // EC information + } + ] + } +} +``` + +Error Cases: +- If creator parameter is invalid, returns error +- If parameters parsing fails, returns error + +Notes: +- Requires admin privileges or valid admin token +- Supports filtering by creators, engine types, statuses, queue name, and EC instances +- Supports cross-cluster filtering \ No newline at end of file diff --git a/docs/modules/index.md b/docs/modules/index.md new file mode 100644 index 0000000000..e5d57a3f12 --- /dev/null +++ b/docs/modules/index.md @@ -0,0 +1,40 @@ +# Apache Linkis Service Module Documentation + +This documentation provides detailed information about each service module in Apache Linkis. Each module contains comprehensive documentation about its APIs, database schemas, and RPC methods. + +## Service Module Categories + +1. [Microservice Governance Services](./microservice-governance/README.md) + - Infrastructure services for the microservices architecture + +2. [Computation Governance Services](./computation-governance/README.md) + - Core services for computation task lifecycle management + +3. [Public Enhancement Services](./public-enhancements/README.md) + - Shared services that provide common capabilities + +## Documentation Standards + +Each service module documentation follows these standards: + +### API Interface Documentation +- Complete list of RESTful endpoints +- Request/response formats with examples +- Authentication requirements +- Error handling and status codes + +### Database Schema Documentation +- Accurate table structures from linkis_ddl.sql +- Primary and foreign key relationships +- Indexing strategies +- Data types and constraints + +### RPC Method Documentation +- Remote procedure calls provided by the service +- Method signatures and parameters +- Return types and error conditions +- Usage examples + +## Contributing + +This documentation is a living document that should be updated as the codebase evolves. When making changes to any service module, please ensure the corresponding documentation is updated to reflect those changes. \ No newline at end of file diff --git a/docs/modules/microservice-governance/README.md b/docs/modules/microservice-governance/README.md new file mode 100644 index 0000000000..07035f1815 --- /dev/null +++ b/docs/modules/microservice-governance/README.md @@ -0,0 +1,112 @@ +# Microservice Governance Services + +The microservice governance services provide the infrastructure foundation for the Linkis microservices architecture. + +## Service Modules + +- [Eureka Service](./eureka.md) - Service registry and discovery center +- [Gateway Service](./gateway.md) - API gateway for request routing and security + +## Overview + +These services form the infrastructure layer of Linkis, providing essential capabilities for service discovery, API routing, and inter-service communication. + +## Common Features + +### Service Discovery +- Service registration and deregistration +- Health checking of services +- Service instance management +- Load balancing support + +### API Gateway +- Request routing and filtering +- Authentication and authorization +- Rate limiting and traffic control +- Request/response transformation + +### Inter-Service Communication +- RESTful service communication +- Load balancing between services +- Circuit breaker pattern implementation +- Service monitoring and metrics + +## API Interface Summary + +### Eureka Service APIs +- Service registration: `POST /eureka/apps/{appName}` +- Service discovery: `GET /eureka/apps/{appName}` +- Health check: `GET /eureka/apps/{appName}/{instanceId}` + +### Gateway Service APIs +- Route management: `GET /actuator/gateway/routes` +- Health check: `GET /actuator/health` +- Gateway metrics: `GET /actuator/metrics` + +## Database Schema Summary + +### Service Registry Table +```sql +CREATE TABLE linkis_service_registry ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + service_name VARCHAR(128) NOT NULL, + instance_id VARCHAR(128) NOT NULL UNIQUE, + instance_address VARCHAR(128), + instance_port INT, + status VARCHAR(50) DEFAULT 'UP', + metadata JSON, + register_time DATETIME DEFAULT CURRENT_TIMESTAMP, + last_heartbeat DATETIME, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Route Table +```sql +CREATE TABLE linkis_gateway_route ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL UNIQUE, + route_order INT DEFAULT 0, + uri VARCHAR(255) NOT NULL, + predicates JSON, + filters JSON, + metadata JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Access Log Table +```sql +CREATE TABLE linkis_gateway_access_log ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + client_ip VARCHAR(50), + request_method VARCHAR(10), + request_uri VARCHAR(500), + request_params TEXT, + user_token VARCHAR(255), + service_id VARCHAR(128), + response_status INT, + response_time BIGINT, + access_time DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +## RPC Methods Summary + +### Eureka Service RPCs +- `registerService(ServiceRegistrationRequest request)` +- `unregisterService(String serviceId, String instanceId)` +- `getServiceInstances(String serviceName)` +- `heartbeat(String serviceId, String instanceId)` + +### Gateway Service RPCs +- `addRoute(GatewayRoute route)` +- `removeRoute(String routeId)` +- `updateRoute(GatewayRoute route)` +- `getRoutes()` +- `configureAuthentication(AuthenticationConfig config)` +- `validateToken(String token)` +- `getUserPermissions(String user)` +- `applyRateLimit(RateLimitConfig config)` +- `getRateLimitStatus(String clientId)` \ No newline at end of file diff --git a/docs/modules/microservice-governance/eureka.md b/docs/modules/microservice-governance/eureka.md new file mode 100644 index 0000000000..1345517e4a --- /dev/null +++ b/docs/modules/microservice-governance/eureka.md @@ -0,0 +1,131 @@ +# Eureka Service + +The Eureka service provides service registration and discovery capabilities for the Linkis microservices architecture. + +## Overview + +This service implements the Eureka server for service discovery, managing the registration, discovery, and health checking of all microservice instances in the Linkis system. + +## Key Components + +### Core Classes +- `SpringCloudEurekaApplication` - Main application class +- Eureka server configuration +- Health check endpoints +- Service registry management + +### Features +- Service instance registration +- Service discovery for clients +- Health status monitoring +- REST API for service management + +## API Interfaces + +### Service Registration +``` +POST /eureka/apps/{appName} +``` + +Request Body: +```xml + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + +``` + +### Service Discovery +``` +GET /eureka/apps/{appName} +``` + +Response: +```xml + + APP-NAME + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + + +``` + +### Health Check +``` +GET /eureka/apps/{appName}/{instanceId} +``` + +Response: +```xml + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + 1234567890 + +``` + +## Database Table Structures + +The Eureka service typically doesn't directly manage database tables, as it stores service registry information in memory. However, it may interact with the following tables for persistent storage: + +### Service Registry Table +```sql +CREATE TABLE linkis_service_registry ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + service_name VARCHAR(128) NOT NULL, + instance_id VARCHAR(128) NOT NULL UNIQUE, + instance_address VARCHAR(128), + instance_port INT, + status VARCHAR(50) DEFAULT 'UP', + metadata JSON, + register_time DATETIME DEFAULT CURRENT_TIMESTAMP, + last_heartbeat DATETIME, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +## RPC Methods + +The Eureka service provides RPC methods for service management: + +### Service Management RPCs + +#### registerService +Registers a service instance: +```java +void registerService(ServiceRegistrationRequest request) +``` + +#### unregisterService +Unregisters a service instance: +```java +void unregisterService(String serviceId, String instanceId) +``` + +#### getServiceInstances +Retrieves instances of a service: +```java +List getServiceInstances(String serviceName) +``` + +#### heartbeat +Sends a heartbeat for a service instance: +```java +void heartbeat(String serviceId, String instanceId) +``` + +## Dependencies + +- Spring Cloud Netflix Eureka Server +- Spring Boot +- Netflix Eureka Core \ No newline at end of file diff --git a/docs/modules/microservice-governance/gateway.md b/docs/modules/microservice-governance/gateway.md new file mode 100644 index 0000000000..4d07a65e88 --- /dev/null +++ b/docs/modules/microservice-governance/gateway.md @@ -0,0 +1,421 @@ +# Gateway Service + +The Gateway service provides API gateway functionality for the Linkis system, routing requests to appropriate backend services and providing security, rate limiting, and other cross-cutting concerns. + +## Overview + +This service implements an API gateway that serves as the single entry point for all client requests to the Linkis system. It handles request routing, authentication, authorization, rate limiting, and other infrastructure concerns. + +## Key Components + +### Core Classes +- `LinkisGatewayApplication` - Main application class +- Route configuration management +- Request/response filtering +- Authentication handling +- Rate limiting implementation + +### Features +- Request routing and load balancing +- Authentication and authorization +- Rate limiting and traffic control +- Request/response transformation +- SSL/TLS termination +- Logging and monitoring + +## API Interfaces + +### Route Management +``` +GET /actuator/gateway/routes +``` + +Response: +```json +{ + "routes": [ + { + "route_id": "linkis-entrance", + "uri": "lb://linkis-entrance", + "predicates": [ + "Path=/api/entrance/**" + ], + "filters": [ + "StripPrefix=1" + ] + } + ] +} +``` + +### Health Check +``` +GET /actuator/health +``` + +Response: +```json +{ + "status": "UP", + "components": { + "discoveryComposite": { + "status": "UP" + }, + "gateway": { + "status": "UP" + } + } +} +``` + +### Gateway Metrics +``` +GET /actuator/metrics +``` + +Response: +```json +{ + "names": [ + "gateway.requests", + "jvm.memory.used", + "http.server.requests" + ] +} +``` + +### Authentication Token Management +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 0, + "list": [], + "pageNum": 1, + "pageSize": 10, + "size": 0, + "startRow": 0, + "endRow": 0, + "pages": 0, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [] + } + } +} +``` + +### Add Authentication Token +``` +POST /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Request Body: +```json +{ + "tokenName": "test-token", + "legalUsers": "*", + "businessOwner": "BDP" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Update Authentication Token +``` +PUT /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Request Body: +```json +{ + "id": 1, + "tokenName": "test-token", + "legalUsers": "user1,user2", + "businessOwner": "BDP" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Get Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/{id} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "tokenName": "test-token", + "legalUsers": "user1,user2", + "businessOwner": "BDP", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + } +} +``` + +### Remove Authentication Token +``` +DELETE /api/rest_j/v1/basedata-manager/gateway-auth-token/{id} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Check Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/checkToken +``` + +Parameters: +- `token`: Authentication token to check (required) +- `checkName`: User name to check (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Decrypt Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/decrypt-token +``` + +Parameters: +- `token`: Authentication token to decrypt (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": "decrypted-token" + } +} +``` + +## Database Table Structures + +The Gateway service manages the following database tables: + +### Gateway Route Table +```sql +CREATE TABLE linkis_gateway_route ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL UNIQUE, + route_order INT DEFAULT 0, + uri VARCHAR(255) NOT NULL, + predicates JSON, + filters JSON, + metadata JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Filter Table +```sql +CREATE TABLE linkis_gateway_filter ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL, + filter_name VARCHAR(128) NOT NULL, + filter_order INT DEFAULT 0, + args JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + FOREIGN KEY (route_id) REFERENCES linkis_gateway_route(route_id) ON DELETE CASCADE +); +``` + +### Authentication Configuration Table +```sql +CREATE TABLE linkis_gateway_auth ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + path_pattern VARCHAR(255) NOT NULL, + auth_required BOOLEAN DEFAULT TRUE, + allowed_roles JSON, + rate_limit INT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Access Log Table +```sql +CREATE TABLE linkis_gateway_access_log ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + client_ip VARCHAR(50), + request_method VARCHAR(10), + request_uri VARCHAR(500), + request_params TEXT, + user_token VARCHAR(255), + service_id VARCHAR(128), + response_status INT, + response_time BIGINT, + access_time DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### Gateway Auth Token Table +```sql +CREATE TABLE `linkis_gateway_auth_token` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `token_name` varchar(255) COLLATE utf8_bin NOT NULL, + `legal_users` varchar(255) COLLATE utf8_bin NOT NULL, + `create_by` varchar(255) COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `elapse_day` bigint(20) DEFAULT '-1', + `update_by` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `business_owner` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `token_alias` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `token_sign` varchar(255) COLLATE utf8_bin DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_token_name` (`token_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; +``` + +## RPC Methods + +The Gateway service provides RPC methods for gateway management: + +### Route Management RPCs + +#### addRoute +Adds a new route configuration: +```java +void addRoute(GatewayRoute route) +``` + +#### removeRoute +Removes a route configuration: +```java +void removeRoute(String routeId) +``` + +#### updateRoute +Updates a route configuration: +```java +void updateRoute(GatewayRoute route) +``` + +#### getRoutes +Retrieves all route configurations: +```java +List getRoutes() +``` + +### Authentication RPCs + +#### configureAuthentication +Configures authentication for a path: +```java +void configureAuthentication(AuthenticationConfig config) +``` + +#### validateToken +Validates an authentication token: +```java +TokenValidationResult validateToken(String token) +``` + +#### getUserPermissions +Retrieves user permissions: +```java +UserPermissions getUserPermissions(String user) +``` + +### Rate Limiting RPCs + +#### applyRateLimit +Applies rate limiting to a route: +```java +void applyRateLimit(RateLimitConfig config) +``` + +#### getRateLimitStatus +Retrieves current rate limit status: +```java +RateLimitStatus getRateLimitStatus(String clientId) +``` + +## Dependencies + +- Spring Cloud Gateway +- Spring Boot +- Spring Security +- Spring Cloud LoadBalancer +- linkis-common +- linkis-httpclient +- Various Spring Cloud components + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- GatewayAuthTokenRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/GatewayAuthTokenRestfulApi.java` + +### MyBatis XML Files +- GatewayRouteMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayRouteMapper.xml` +- GatewayFilterMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayFilterMapper.xml` +- GatewayAuthMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAuthMapper.xml` +- GatewayAccessLogMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAccessLogMapper.xml` +- GatewayAuthTokenMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAuthTokenMapper.xml` \ No newline at end of file diff --git a/docs/modules/microservice-governance/monitor.md b/docs/modules/microservice-governance/monitor.md new file mode 100644 index 0000000000..1a52b1696b --- /dev/null +++ b/docs/modules/microservice-governance/monitor.md @@ -0,0 +1,261 @@ +# Monitor Service + +Monitor service is responsible for monitoring the health status of various components in the Linkis system, including resource monitoring, node heartbeat monitoring, etc. + +## Table of Contents +- [API Interfaces](#api-interfaces) +- [Database Tables](#database-tables) +- [RPC Methods](#rpc-methods) +- [Interface Classes and MyBatis XML Files](#interface-classes-and-mybatis-xml-files) + +## API Interfaces + +### Get Application List +**POST /linkisManager/rm/applicationlist** + +Get the list of applications for a specific user. + +Request Parameters: +```json +{ + "userCreator": "string", + "engineType": "string" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/applicationlist", + "status": 0, + "message": "OK", + "data": { + "applications": [] + } +} +``` + +### Reset User Resource +**DELETE /linkisManager/rm/resetResource** + +Reset user resource, admin only. + +Request Parameters: +- resourceId (optional): Integer + +Response: +```json +{ + "method": "/api/linkisManager/rm/resetResource", + "status": 0, + "message": "success", + "data": {} +} +``` + +### List All Engine Types +**GET /linkisManager/rm/engineType** + +Get all supported engine types. + +Response: +```json +{ + "method": "/api/linkisManager/rm/engineType", + "status": 0, + "message": "OK", + "data": { + "engineType": ["string"] + } +} +``` + +### Get All User Resources +**GET /linkisManager/rm/allUserResource** + +Get all user resources, admin only. + +Request Parameters: +- username (optional): String +- creator (optional): String +- engineType (optional): String +- page (optional): Integer +- size (optional): Integer + +Response: +```json +{ + "method": "/api/linkisManager/rm/allUserResource", + "status": 0, + "message": "OK", + "data": { + "resources": [], + "total": 0 + } +} +``` + +### Get User Resource by Label +**GET /linkisManager/rm/get-user-resource** + +Get user resource by label. + +Request Parameters: +- username: String +- creator: String +- engineType: String + +Response: +```json +{ + "method": "/api/linkisManager/rm/get-user-resource", + "status": 0, + "message": "OK", + "data": { + "resources": [] + } +} +``` + +### Get User Resources +**POST /linkisManager/rm/userresources** + +Get user resources. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/userresources", + "status": 0, + "message": "OK", + "data": { + "userResources": [] + } +} +``` + +### Get Engines +**POST /linkisManager/rm/engines** + +Get engines for a user. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/engines", + "status": 0, + "message": "OK", + "data": { + "engines": [] + } +} +``` + +### Get Queue Resource +**POST /linkisManager/rm/queueresources** + +Get queue resource information. + +Request Parameters: +```json +{ + "queuename": "string", + "clustername": "string", + "clustertype": "string", + "crossCluster": "boolean" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/queueresources", + "status": 0, + "message": "OK", + "data": { + "queueInfo": {}, + "userResources": [] + } +} +``` + +### Get Queues +**POST /linkisManager/rm/queues** + +Get queue information. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/queues", + "status": 0, + "message": "OK", + "data": { + "queues": [] + } +} +``` + +## Database Tables + +### linkis_cg_rm_external_resource_provider +```sql +CREATE TABLE `linkis_cg_rm_external_resource_provider` ( + `id` int(10) NOT NULL AUTO_INCREMENT, + `resource_type` varchar(32) NOT NULL, + `name` varchar(32) NOT NULL, + `labels` varchar(32) DEFAULT NULL, + `config` text NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### linkis_cg_rm_resource_action_record +```sql +CREATE TABLE linkis_cg_rm_resource_action_record ( + `id` INT(20) NOT NULL AUTO_INCREMENT, + `label_value` VARCHAR(100) NOT NULL, + `ticket_id` VARCHAR(100) NOT NULL, + `request_times` INT(8), + `request_resource_all` VARCHAR(100), + `used_times` INT(8), + `used_resource_all` VARCHAR(100), + `release_times` INT(8), + `release_resource_all` VARCHAR(100), + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `label_value_ticket_id` (`label_value`, `ticket_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` + +## RPC Methods + +Monitor service does not expose specific RPC methods directly. It primarily works through the RESTful APIs listed above. + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- RMMonitorRest.scala: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\rm\restful\RMMonitorRest.scala` + +### MyBatis XML Files +The Monitor service uses the following persistence layer interfaces which may have corresponding MyBatis XML files: +- LabelManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\LabelManagerPersistence.scala` +- ResourceManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\ResourceManagerPersistence.scala` +- NodeManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\NodeManagerPersistence.scala` +- NodeMetricManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\NodeMetricManagerPersistence.scala` \ No newline at end of file diff --git a/docs/modules/public-enhancements/README.md b/docs/modules/public-enhancements/README.md new file mode 100644 index 0000000000..03433e3363 --- /dev/null +++ b/docs/modules/public-enhancements/README.md @@ -0,0 +1,235 @@ +# Public Enhancement Services + +The public enhancement services provide shared capabilities used across the Linkis platform. + +## Service Modules + +- [Public Service](./publicservice.md) - Core public services +- [Configuration Service](./configuration.md) - Configuration management +- [BML Service](./bml.md) - Big Data Material Library +- [DataSource Service](./datasource.md) - Data source management +- [Context Service](./context.md) - Context and variable sharing +- [Monitor Service](./monitor.md) - System monitoring + +## Overview + +These services provide common capabilities that are used across the Linkis platform, including file management, configuration management, data source management, context sharing, and system monitoring. + +## Common Features + +### Resource Management +- Binary and material management +- User-defined function management +- Shared resource tracking + +### Configuration Management +- Centralized configuration service +- Runtime configuration management +- Configuration versioning + +### Context Management +- Cross-application context sharing +- Variable and parameter management +- Unified context service + +### Data Source Management +- Data source registration and management +- Metadata querying +- Connection testing and validation + +### System Monitoring +- Performance metrics collection +- System health monitoring +- Alerting and notifications + +## API Interface Summary + +### Public Service APIs +- File system operations +- Variable management +- Error code querying + +### Configuration Service APIs +- Configuration retrieval: `GET /api/rest_j/v1/configuration` +- Configuration update: `POST /api/rest_j/v1/configuration/update` +- Template management: `GET /api/rest_j/v1/configuration/template` + +### BML Service APIs +- File upload: `POST /api/rest_j/v1/bml/upload` +- File download: `GET /api/rest_j/v1/bml/download` +- File version list: `GET /api/rest_j/v1/bml/versions` + +### DataSource Service APIs +- Data source CRUD: `POST/GET/PUT/DELETE /api/rest_j/v1/datasource` +- Metadata query: `GET /api/rest_j/v1/datasource/metadata` +- Connection test: `POST /api/rest_j/v1/datasource/connect` + +### Context Service APIs +- Context creation: `POST /api/rest_j/v1/context` +- Variable management: `POST /api/rest_j/v1/context/variable` +- Context sharing: `POST /api/rest_j/v1/context/share` + +### Monitor Service APIs +- Metrics collection: `GET /api/rest_j/v1/monitor/metrics` +- Health check: `GET /api/rest_j/v1/monitor/health` +- Alert management: `POST /api/rest_j/v1/monitor/alert` + +## Database Schema Summary + +### BML Resources Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'resource uuid', + `is_private` TINYINT(1) DEFAULT 0 COMMENT 'Whether the resource is private, 0 means private, 1 means public', + `resource_header` TINYINT(1) DEFAULT 0 COMMENT 'Classification, 0 means unclassified, 1 means classified', + `downloaded_file_name` varchar(200) DEFAULT NULL COMMENT 'File name when downloading', + `sys` varchar(100) NOT NULL COMMENT 'Owning system', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Created time', + `owner` varchar(200) NOT NULL COMMENT 'Resource owner', + `is_expire` TINYINT(1) DEFAULT 0 COMMENT 'Whether expired, 0 means not expired, 1 means expired', + `expire_type` varchar(50) DEFAULT null COMMENT 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time', + `expire_time` varchar(50) DEFAULT null COMMENT 'Expiration time, one day by default', + `max_version` int(20) DEFAULT 10 COMMENT 'The default is 10, which means to keep the latest 10 versions', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Updated time', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_eflag`(`resource_id`, `enable_flag`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Version Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `file_md5` varchar(32) NOT NULL COMMENT 'Md5 summary of the file', + `version` varchar(20) NOT NULL COMMENT 'Resource version (v plus five digits)', + `size` int(10) NOT NULL COMMENT 'File size', + `start_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `end_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `resource` varchar(2000) NOT NULL COMMENT 'Resource content (file information including path and file name)', + `description` varchar(2000) DEFAULT NULL COMMENT 'description', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Started time', + `end_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Stoped time', + `client_ip` varchar(200) NOT NULL COMMENT 'Client ip', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_version`(`resource_id`, `version`), + PRIMARY KEY (`id`) +); +``` + +### Configuration Key Table +```sql +CREATE TABLE `linkis_ps_configuration_config_key`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Table +```sql +CREATE TABLE `linkis_ps_datasource_table` ( + `id` bigint(255) NOT NULL AUTO_INCREMENT, + `database` varchar(64) COLLATE utf8_bin NOT NULL, + `name` varchar(64) COLLATE utf8_bin NOT NULL, + `alias` varchar(64) COLLATE utf8_bin DEFAULT NULL, + `creator` varchar(16) COLLATE utf8_bin NOT NULL, + `comment` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `create_time` datetime NOT NULL, + `product_name` varchar(64) COLLATE utf8_bin DEFAULT NULL, + `project_name` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `usage` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `lifecycle` int(4) NOT NULL, + `use_way` int(4) NOT NULL, + `is_import` tinyint(1) NOT NULL, + `model_level` int(4) NOT NULL, + `is_external_use` tinyint(1) NOT NULL, + `is_partition_table` tinyint(1) NOT NULL, + `is_available` tinyint(1) NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_db_name` (`database`,`name`) +); +``` + +### Context Map Table +```sql +CREATE TABLE `linkis_ps_cs_context_map` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(128) DEFAULT NULL, + `context_scope` varchar(32) DEFAULT NULL, + `context_type` varchar(32) DEFAULT NULL, + `props` text, + `value` mediumtext, + `context_id` int(11) DEFAULT NULL, + `keywords` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`), + KEY `idx_keywords` (`keywords`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods Summary + +### Public Service RPCs +- `getErrorCode(String errorCode)` +- `getVariable(String variableName)` +- `setVariable(String variableName, String value)` + +### Configuration Service RPCs +- `getConfiguration(String user, String creator, String engineType)` +- `updateConfiguration(String user, ConfigurationUpdateRequest request)` +- `getTemplateConfiguration(String engineType)` + +### BML Service RPCs +- `uploadResource(ResourceUploadRequest request)` +- `downloadResource(String resourceId, String version)` +- `deleteResource(String resourceId)` +- `getResourceInfo(String resourceId)` + +### DataSource Service RPCs +- `createDataSource(DataSourceCreateRequest request)` +- `updateDataSource(DataSourceUpdateRequest request)` +- `deleteDataSource(Long dataSourceId)` +- `queryDataSource(DataSourceQueryRequest request)` + +### Context Service RPCs +- `createContext(ContextCreateRequest request)` +- `getContextValue(String contextId, String key)` +- `setContextValue(String contextId, String key, String value)` +- `removeContext(String contextId)` + +### Monitor Service RPCs +- `collectMetrics(MetricsCollectionRequest request)` +- `getHealthStatus()` +- `sendAlert(AlertRequest request)` + +## Dependencies + +- linkis-commons - Shared utilities +- linkis-protocol - Communication protocols +- linkis-rpc - Remote procedure calls +- Various database drivers +- Spring Cloud ecosystem \ No newline at end of file diff --git a/docs/modules/public-enhancements/bml.md b/docs/modules/public-enhancements/bml.md new file mode 100644 index 0000000000..6f5673e09a --- /dev/null +++ b/docs/modules/public-enhancements/bml.md @@ -0,0 +1,634 @@ +# BML Service + +The BML (Big Data Material Library) Service provides file and material management capabilities for the Linkis system. + +## Overview + +This service manages the storage, versioning, and sharing of files and materials used in big data processing tasks. + +## Key Components + +### Core Classes +- `LinkisBMLApplication` - Main application class +- File upload and download +- File version management +- File sharing and access control + +### Features +- File upload and download +- File versioning +- File sharing +- Access control +- File metadata management + +## API Interfaces + +### File Upload +``` +POST /api/rest_j/v1/bml/upload +``` + +Parameters: +- `system` (optional): System name +- `resourceHeader` (optional): Resource header +- `isExpire` (optional): Whether resource expires +- `expireType` (optional): Expiration type +- `expireTime` (optional): Expiration time +- `maxVersion` (optional): Maximum version count +- `file` (required): File to upload + +Response: +``` +{ + "method": "/api/bml/upload", + "status": 0, + "message": "The task of submitting and uploading resources was successful(提交上传资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000001", + "taskId": 12345 + } +} +``` + +### File Download +``` +GET /api/rest_j/v1/bml/download +``` + +Parameters: +- `resourceId`: Resource ID to download (required) +- `version`: Version to download (optional, defaults to latest) + +Response: +``` +Binary file content +``` + +### File Version List +``` +GET /api/rest_j/v1/bml/getVersions +``` + +Parameters: +- `resourceId`: Resource ID to list versions for (required) +- `currentPage`: Current page number (optional) +- `pageSize`: Page size (optional) + +Response: +```json +{ + "method": "/api/bml/getVersions", + "status": 0, + "message": "Version information obtained successfully (成功获取版本信息)", + "data": { + "ResourceVersions": { + "resourceId": "resource-12345", + "user": "testuser", + "versions": [ + { + "version": "v000001", + "size": 1024, + "createTime": "2023-01-01 12:00:00" + } + ] + } + } +} +``` + +### File Update +``` +POST /api/rest_j/v1/bml/updateVersion +``` + +Parameters: +- `resourceId`: Resource ID to update (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/updateVersion", + "status": 0, + "message": "The update resource task was submitted successfully(提交更新资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000002", + "taskId": 12346 + } +} +``` + +### File Delete +``` +POST /api/rest_j/v1/bml/deleteResource +``` + +Request Body: +```json +{ + "resourceId": "resource-12345" +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteResource", + "status": 0, + "message": "Resource deleted successfully(删除资源成功)" +} +``` + +### Batch File Delete +``` +POST /api/rest_j/v1/bml/deleteResources +``` + +Request Body: +```json +{ + "resourceIds": ["resource-12345", "resource-12346"] +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteResources", + "status": 0, + "message": "Batch deletion of resource was successful(批量删除资源成功)" +} +``` + +### File Information +``` +GET /api/rest_j/v1/bml/getBasic +``` + +Parameters: +- `resourceId`: Resource ID to get information for (required) + +Response: +```json +{ + "method": "/api/bml/getBasic", + "status": 0, + "message": "Acquisition of resource basic information successfully(获取资源基本信息成功)", + "data": { + "basic": { + "resourceId": "resource-12345", + "owner": "testuser", + "createTime": "2023-01-01 12:00:00", + "downloadedFileName": "test.csv", + "expireTime": "Resource not expired(资源不过期)", + "numberOfVerions": 10 + } + } +} +``` + +### Version Delete +``` +POST /api/rest_j/v1/bml/deleteVersion +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "version": "v000001" +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteVersion", + "status": 0, + "message": "Deleted version successfully(删除版本成功)" +} +``` + +### Change Owner +``` +POST /api/rest_j/v1/bml/changeOwner +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "oldOwner": "testuser", + "newOwner": "newuser" +} +``` + +Response: +```json +{ + "method": "/api/bml/changeOwner", + "status": 0, + "message": "更新owner成功！" +} +``` + +### Copy Resource To Another User +``` +POST /api/rest_j/v1/bml/copyResourceToAnotherUser +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "anotherUser": "newuser" +} +``` + +Response: +```json +{ + "method": "/api/bml/copyResourceToAnotherUser", + "status": 0, + "message": "success", + "data": { + "resourceId": "resource-67890" + } +} +``` + +### Rollback Version +``` +POST /api/rest_j/v1/bml/rollbackVersion +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "version": "v000001" +} +``` + +Response: +```json +{ + "method": "/api/bml/rollbackVersion", + "status": 0, + "message": "success", + "data": { + "resourceId": "resource-12345", + "version": "v000001" + } +} +``` + +### Create BML Project +``` +POST /api/rest_j/v1/bml/createBmlProject +``` + +Request Body: +```json +{ + "projectName": "test-project", + "editUsers": ["user1", "user2"], + "accessUsers": ["user3", "user4"] +} +``` + +Response: +```json +{ + "method": "/api/bml/createBmlProject", + "status": 0, + "message": "success to create project(创建工程ok)" +} +``` + +### Upload Share Resource +``` +POST /api/rest_j/v1/bml/uploadShareResource +``` + +Parameters: +- `system` (optional): System name +- `resourceHeader` (optional): Resource header +- `isExpire` (optional): Whether resource expires +- `expireType` (optional): Expiration type +- `expireTime` (optional): Expiration time +- `maxVersion` (optional): Maximum version count +- `projectName`: Project name (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/uploadShareResource", + "status": 0, + "message": "The task of submitting and uploading resources was successful(提交上传资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000001", + "taskId": 12345 + } +} +``` + +### Update Share Resource +``` +POST /api/rest_j/v1/bml/updateShareResource +``` + +Parameters: +- `resourceId`: Resource ID to update (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/updateShareResource", + "status": 0, + "message": "The update resource task was submitted successfully(提交更新资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000002", + "taskId": 12346 + } +} +``` + +### Download Share Resource +``` +GET /api/rest_j/v1/bml/downloadShareResource +``` + +Parameters: +- `resourceId`: Resource ID to download (required) +- `version`: Version to download (optional, defaults to latest) + +Response: +``` +Binary file content +``` + +### Update Project Users +``` +POST /api/rest_j/v1/bml/updateProjectUsers +``` + +Request Body: +```json +{ + "projectName": "test-project", + "editUsers": ["user1", "user2", "user5"], + "accessUsers": ["user3", "user4", "user6"] +} +``` + +Response: +```json +{ + "method": "/api/bml/updateProjectUsers", + "status": 0, + "message": "Updated project related user success(更新工程的相关用户成功)" +} +``` + +## Database Table Structures + +The BML Service uses the following database tables from linkis_ddl.sql: + +### BML Resources Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'resource uuid', + `is_private` TINYINT(1) DEFAULT 0 COMMENT 'Whether the resource is private, 0 means private, 1 means public', + `resource_header` TINYINT(1) DEFAULT 0 COMMENT 'Classification, 0 means unclassified, 1 means classified', + `downloaded_file_name` varchar(200) DEFAULT NULL COMMENT 'File name when downloading', + `sys` varchar(100) NOT NULL COMMENT 'Owning system', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Created time', + `owner` varchar(200) NOT NULL COMMENT 'Resource owner', + `is_expire` TINYINT(1) DEFAULT 0 COMMENT 'Whether expired, 0 means not expired, 1 means expired', + `expire_type` varchar(50) DEFAULT null COMMENT 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time', + `expire_time` varchar(50) DEFAULT null COMMENT 'Expiration time, one day by default', + `max_version` int(20) DEFAULT 10 COMMENT 'The default is 10, which means to keep the latest 10 versions', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Updated time', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_eflag`(`resource_id`, `enable_flag`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Version Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `file_md5` varchar(32) NOT NULL COMMENT 'Md5 summary of the file', + `version` varchar(20) NOT NULL COMMENT 'Resource version (v plus five digits)', + `size` int(10) NOT NULL COMMENT 'File size', + `start_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `end_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `resource` varchar(2000) NOT NULL COMMENT 'Resource content (file information including path and file name)', + `description` varchar(2000) DEFAULT NULL COMMENT 'description', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Started time', + `end_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Stoped time', + `client_ip` varchar(200) NOT NULL COMMENT 'Client ip', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_version`(`resource_id`, `version`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Permission Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_permission` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `permission` varchar(10) NOT NULL COMMENT 'permission', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'created time', + `system` varchar(50) default "dss" COMMENT 'creator', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'updated time', + `updator` varchar(50) NOT NULL COMMENT 'updator', + PRIMARY KEY (`id`) +); +``` + +### BML Resources Task Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_task` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `resource_id` varchar(50) DEFAULT NULL COMMENT 'resource uuid', + `version` varchar(20) DEFAULT NULL COMMENT 'Resource version number of the current operation', + `operation` varchar(20) NOT NULL COMMENT 'Operation type. upload = 0, update = 1', + `state` varchar(20) NOT NULL DEFAULT 'Schduled' COMMENT 'Current status of the task:Schduled, Running, Succeed, Failed,Cancelled', + `submit_user` varchar(20) NOT NULL DEFAULT '' COMMENT 'Job submission user name', + `system` varchar(20) DEFAULT 'dss' COMMENT 'Subsystem name: wtss', + `instance` varchar(128) NOT NULL COMMENT 'Material library example', + `client_ip` varchar(50) DEFAULT NULL COMMENT 'Request IP', + `extra_params` text COMMENT 'Additional key information. Such as the resource IDs and versions that are deleted in batches, and all versions under the resource are deleted', + `err_msg` varchar(2000) DEFAULT NULL COMMENT 'Task failure information.e.getMessage', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Starting time', + `end_time` datetime DEFAULT NULL COMMENT 'End Time', + `last_update_time` datetime NOT NULL COMMENT 'Last update time', + unique key `uniq_rid_version` (resource_id, version), + PRIMARY KEY (`id`) +); +``` + +### BML Project Table +```sql +create table if not exists linkis_ps_bml_project( + `id` int(10) NOT NULL AUTO_INCREMENT, + `name` varchar(128) DEFAULT NULL, + `system` varchar(64) not null default "dss", + `source` varchar(1024) default null, + `description` varchar(1024) default null, + `creator` varchar(128) not null, + `enabled` tinyint default 1, + `create_time` datetime DEFAULT now(), + unique key `uniq_name` (`name`), +PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The BML Service provides several RPC methods for file management: + +### Resource RPCs + +#### uploadResource +Uploads a resource: +```java +ResourceUploadResult uploadResource(ResourceUploadRequest request) +``` + +#### downloadResource +Downloads a resource: +```java +ResourceContent downloadResource(String resourceId, String version) +``` + +#### deleteResource +Deletes a resource: +```java +void deleteResource(String resourceId) +``` + +#### getResourceInfo +Retrieves resource information: +```java +ResourceInfo getResourceInfo(String resourceId) +``` + +#### listResources +Lists resources for a user: +```java +List listResources(String username) +``` + +### Version RPCs + +#### listVersions +Lists versions of a resource: +```java +List listVersions(String resourceId) +``` + +#### updateResource +Updates a resource with a new version: +```java +ResourceUpdateResult updateResource(ResourceUpdateRequest request) +``` + +#### deleteVersion +Deletes a specific version of a resource: +```java +void deleteVersion(String resourceId, String version) +``` + +#### getVersionInfo +Gets information about a specific version: +```java +ResourceVersion getVersionInfo(String resourceId, String version) +``` + +### Permission RPCs + +#### grantPermission +Grants permission to a user: +```java +void grantPermission(String resourceId, String username, String permission) +``` + +#### checkPermission +Checks if a user has permission: +```java +boolean checkPermission(String resourceId, String username, String permission) +``` + +#### revokePermission +Revokes permission from a user: +```java +void revokePermission(String resourceId, String username) +``` + +#### listPermissions +Lists all permissions for a resource: +```java +List listPermissions(String resourceId) +``` + +### Project RPCs + +#### createProject +Creates a new project: +```java +Project createProject(ProjectCreateRequest request) +``` + +#### deleteProject +Deletes a project: +```java +void deleteProject(Long projectId) +``` + +#### addResourceToProject +Adds a resource to a project: +```java +void addResourceToProject(Long projectId, String resourceId) +``` + +#### removeResourceFromProject +Removes a resource from a project: +```java +void removeResourceFromProject(Long projectId, String resourceId) +``` + +## Dependencies + +- linkis-bml-server +- linkis-mybatis +- linkis-rpc +- linkis-protocol + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- BmlRestfulApi: `linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java` +- BmlProjectRestful: `linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java` +- BMLFsRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/BMLFsRestfulApi.java` + +### MyBatis XML Files +- ResourceMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/ResourceMapper.xml` +- VersionMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/VersionMapper.xml` +- TaskMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/TaskMapper.xml` +- DownloadMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/DownloadMapper.xml` +- BmlProjectMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/BmlProjectMapper.xml` diff --git a/docs/modules/public-enhancements/configuration.md b/docs/modules/public-enhancements/configuration.md new file mode 100644 index 0000000000..01b1e58352 --- /dev/null +++ b/docs/modules/public-enhancements/configuration.md @@ -0,0 +1,1264 @@ +# Configuration Service + +The Configuration Service provides centralized configuration management for the Linkis system. + +## Overview + +This service manages configuration properties for all Linkis components, providing a unified interface for configuration retrieval, updates, and validation. + +## Key Components + +### Core Classes +- `LinkisConfigurationApp` - Main application class +- Configuration management +- Configuration validation +- Template management + +### Features +- Global configuration management +- User-specific configuration +- Configuration validation +- Template-based configuration +- Engine-type specific configuration + +## API Interfaces + +### Configuration Retrieval APIs + +#### Get Full Configuration Trees +``` +GET /api/rest_j/v1/configuration/getFullTreesByAppName +``` + +Parameters: +- `engineType`: Engine type (e.g., spark, hive) - optional +- `version`: Engine version - optional +- `creator`: Creator application - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getFullTreesByAppName", + "status": 0, + "message": "success", + "data": { + "fullTree": [ + { + "name": "JVM Configuration", + "description": "JVM configuration for engine", + "settings": [ + { + "key": "wds.linkis.engineconn.java.driver.memory", + "value": "2g", + "defaultValue": "1g", + "description": "Memory size of driver JVM process", + "validateType": "Regex", + "validateRange": "^[0-9]+(\\.?[0-9]*)([gGmMkK])?$", + "level": 1, + "hidden": false, + "advanced": false + } + ] + } + ] + } +} +``` + +#### Get Configuration Category +``` +GET /api/rest_j/v1/configuration/getCategory +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getCategory", + "status": 0, + "message": "success", + "data": { + "Category": [ + { + "categoryId": 1, + "categoryName": "Engine Resource", + "description": "Engine resource configuration" + } + ] + } +} +``` + +#### Get Configuration Item List +``` +GET /api/rest_j/v1/configuration/getItemList +``` + +Parameters: +- `engineType`: Engine type (e.g., spark, hive) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getItemList", + "status": 0, + "message": "success", + "data": { + "itemList": [ + { + "key": "spark.executor.instances", + "name": "Executor Instances", + "description": "Number of executor instances", + "engineType": "spark", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "boundaryType": 3, + "defaultValue": "1", + "require": 0 + } + ] + } +} +``` + +#### List All Engine Types +``` +GET /api/rest_j/v1/configuration/engineType +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/engineType", + "status": 0, + "message": "success", + "data": { + "engineType": ["spark", "hive", "python"] + } +} +``` + +#### Get Key Value +``` +GET /api/rest_j/v1/configuration/keyvalue +``` + +Parameters: +- `engineType`: Engine type - default "*" +- `version`: Engine version - default "*" +- `creator`: Creator application - default "*" +- `configKey`: Configuration key (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValues": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2g", + "configLabelId": 1 + } + ] + } +} +``` + +#### Get Base Key Value +``` +GET /api/rest_j/v1/configuration/baseKeyValue +``` + +Parameters: +- `engineType`: Engine type - optional +- `key`: Configuration key - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success", + "data": { + "configKeyList": [ + { + "id": 1, + "key": "spark.executor.instances", + "description": "Number of executor instances", + "name": "Executor Instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "engineConnType": "spark", + "isHidden": 0, + "isAdvanced": 0, + "level": 1, + "treeName": "Spark Configuration", + "boundaryType": 3, + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 + } + ], + "totalPage": 10 + } +} +``` + +#### Get User Key Value +``` +GET /api/rest_j/v1/configuration/userKeyValue +``` + +Parameters: +- `engineType`: Engine type - optional +- `key`: Configuration key - optional +- `creator`: Creator application - optional +- `user`: Username - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/userKeyValue", + "status": 0, + "message": "success", + "data": { + "configValueList": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "engineType": "spark", + "key": "spark.executor.instances", + "creator": "IDE", + "user": "testuser" + } + ], + "totalPage": 1 + } +} +``` + +### Configuration Management APIs + +#### Create First Category +``` +POST /api/rest_j/v1/configuration/createFirstCategory +``` + +Request Body: +```json +{ + "categoryName": "Engine Resource", + "description": "Engine resource configuration" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/createFirstCategory", + "status": 0, + "message": "success" +} +``` + +#### Delete Category +``` +POST /api/rest_j/v1/configuration/deleteCategory +``` + +Request Body: +```json +{ + "categoryId": 1 +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/deleteCategory", + "status": 0, + "message": "success" +} +``` + +#### Create Second Category +``` +POST /api/rest_j/v1/configuration/createSecondCategory +``` + +Request Body: +```json +{ + "categoryId": 1, + "engineType": "spark", + "version": "2.4.3", + "description": "Spark 2.4.3 configuration" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/createSecondCategory", + "status": 0, + "message": "success" +} +``` + +#### Update Category Info +``` +POST /api/rest_j/v1/configuration/updateCategoryInfo +``` + +Request Body: +```json +{ + "categoryId": 1, + "description": "Updated description" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/updateCategoryInfo", + "status": 0, + "message": "success" +} +``` + +#### Save Full Tree +``` +POST /api/rest_j/v1/configuration/saveFullTree +``` + +Request Body: +```json +{ + "creator": "IDE", + "engineType": "spark-2.4.3", + "fullTree": [ + { + "name": "JVM Configuration", + "description": "JVM configuration for engine", + "settings": [ + { + "key": "wds.linkis.engineconn.java.driver.memory", + "value": "2g", + "defaultValue": "1g", + "description": "Memory size of driver JVM process", + "validateType": "Regex", + "validateRange": "^[0-9]+(\\.?[0-9]*)([gGmMkK])?$", + "level": 1, + "hidden": false, + "advanced": false + } + ] + } + ] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/saveFullTree", + "status": 0, + "message": "success" +} +``` + +#### Save Key Value +``` +POST /api/rest_j/v1/configuration/keyvalue +``` + +Request Body: +```json +{ + "engineType": "spark", + "version": "2.4.3", + "creator": "IDE", + "configKey": "spark.executor.instances", + "configValue": "2", + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValue": { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1 + } + } +} +``` + +#### Delete Key Value +``` +DELETE /api/rest_j/v1/configuration/keyvalue +``` + +Request Body: +```json +{ + "engineType": "spark", + "version": "2.4.3", + "creator": "IDE", + "configKey": "spark.executor.instances" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValues": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1 + } + ] + } +} +``` + +#### Save Base Key Value +``` +POST /api/rest_j/v1/configuration/baseKeyValue +``` + +Request Body: +```json +{ + "key": "spark.executor.instances", + "name": "Executor Instances", + "description": "Number of executor instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "boundaryType": 3, + "treeName": "Spark Configuration", + "engineType": "spark", + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success", + "data": { + "configKey": { + "id": 1, + "key": "spark.executor.instances", + "description": "Number of executor instances", + "name": "Executor Instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "engineConnType": "spark", + "isHidden": 0, + "isAdvanced": 0, + "level": 1, + "treeName": "Spark Configuration", + "boundaryType": 3, + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 + } + } +} +``` + +#### Delete Base Key Value +``` +DELETE /api/rest_j/v1/configuration/baseKeyValue +``` + +Parameters: +- `id`: Configuration key ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success" +} +``` + +### Template Management APIs + +#### Update Key Mapping +``` +POST /api/rest_j/v1/configuration/template/updateKeyMapping +``` + +Request Body: +```json +{ + "templateUid": "template-uuid", + "templateName": "Spark Template", + "engineType": "spark", + "operator": "admin", + "isFullMode": true, + "itemList": [ + { + "keyId": 1, + "maxValue": "10", + "minValue": "1" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/updateKeyMapping", + "status": 0, + "message": "success" +} +``` + +#### Query Key Info List +``` +POST /api/rest_j/v1/configuration/template/queryKeyInfoList +``` + +Request Body: +```json +{ + "templateUidList": ["template-uuid-1", "template-uuid-2"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/queryKeyInfoList", + "status": 0, + "message": "success", + "data": { + "list": [ + { + "templateName": "Spark Template", + "engineType": "spark", + "configKey": "spark.executor.instances", + "maxValue": "10", + "minValue": "1" + } + ] + } +} +``` + +#### Apply Configuration Template +``` +POST /api/rest_j/v1/configuration/template/apply +``` + +Request Body: +```json +{ + "templateUid": "template-uuid", + "application": "IDE", + "engineType": "spark", + "engineVersion": "2.4.3", + "operator": "admin", + "userList": ["user1", "user2"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/apply", + "status": 0, + "message": "success", + "data": { + "success": true, + "failedUsers": [] + } +} +``` + +#### Encrypt Datasource Password +``` +GET /api/rest_j/v1/configuration/template/encrypt +``` + +Parameters: +- `isEncrypt`: Encrypt flag - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/encrypt", + "status": 0, + "message": "success" +} +``` + +### Tenant Configuration APIs + +#### Create Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/create-tenant +``` + +Request Body: +```json +{ + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1", + "desc": "Test tenant", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/create-tenant", + "status": 0, + "message": "success" +} +``` + +#### Update Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/update-tenant +``` + +Request Body: +```json +{ + "id": 1, + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1-updated", + "desc": "Updated tenant", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/update-tenant", + "status": 0, + "message": "success" +} +``` + +#### Delete Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/delete-tenant +``` + +Parameters: +- `id`: Tenant ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/delete-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Tenant List +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-tenant-list +``` + +Parameters: +- `user`: Username - optional +- `creator`: Creator application - optional +- `tenantValue`: Tenant value - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-tenant-list", + "status": 0, + "message": "success", + "data": { + "tenantList": [ + { + "id": 1, + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1", + "desc": "Test tenant", + "bussinessUser": "admin", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Check User Creator +``` +GET /api/rest_j/v1/configuration/tenant-mapping/check-user-creator +``` + +Parameters: +- `user`: Username (required) +- `creator`: Creator application (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/check-user-creator", + "status": 0, + "message": "success", + "data": { + "exist": true + } +} +``` + +#### Save Department Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/save-department-tenant +``` + +Request Body: +```json +{ + "creator": "IDE", + "department": "Engineering", + "departmentId": "dept1", + "tenantValue": "tenant1", + "createBy": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/save-department-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Department Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-department-tenant +``` + +Parameters: +- `departmentId`: Department ID - optional +- `department`: Department name - optional +- `creator`: Creator application - optional +- `tenantValue`: Tenant value - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-department-tenant", + "status": 0, + "message": "success", + "data": { + "tenantList": [ + { + "id": 1, + "creator": "IDE", + "department": "Engineering", + "departmentId": "dept1", + "tenantValue": "tenant1", + "createBy": "admin", + "isValid": "Y", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Delete Department Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/delete-department-tenant +``` + +Parameters: +- `id`: Department tenant ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/delete-department-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Department List +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-department +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-department", + "status": 0, + "message": "success", + "data": { + "departmentList": ["Engineering", "Marketing", "Sales"] + } +} +``` + +#### Query User Department +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-user-department +``` + +Parameters: +- `username`: Username (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-user-department", + "status": 0, + "message": "success", + "data": { + "department": "Engineering" + } +} +``` + +### User IP Configuration APIs + +#### Create User IP +``` +POST /api/rest_j/v1/configuration/user-ip-mapping/create-user-ip +``` + +Request Body: +```json +{ + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2", + "desc": "Allowed IPs for testuser", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/create-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Update User IP +``` +POST /api/rest_j/v1/configuration/user-ip-mapping/update-user-ip +``` + +Request Body: +```json +{ + "id": 1, + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2,192.168.1.3", + "desc": "Updated allowed IPs for testuser", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/update-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Delete User IP +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/delete-user-ip +``` + +Parameters: +- `id`: User IP ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/delete-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Query User IP List +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/query-user-ip-list +``` + +Parameters: +- `user`: Username - optional +- `creator`: Creator application - optional +- `pageNow`: Page number (required) +- `pageSize`: Page size (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/query-user-ip-list", + "status": 0, + "message": "success", + "data": { + "userIpList": [ + { + "id": 1, + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2", + "desc": "Allowed IPs for testuser", + "bussinessUser": "admin", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Check User Creator +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/check-user-creator +``` + +Parameters: +- `user`: Username (required) +- `creator`: Creator application (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/check-user-creator", + "status": 0, + "message": "success", + "data": { + "exist": true + } +} +``` + +## Database Table Structures + +The Configuration Service uses the following database tables from linkis_ddl.sql: + +### Configuration Config Key Table +```sql +CREATE TABLE `linkis_ps_configuration_config_key`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) +); +``` + +### Configuration Key Engine Relation Table +```sql +CREATE TABLE `linkis_ps_configuration_key_engine_relation`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `config_key_id` bigint(20) NOT NULL COMMENT 'config key id', + `engine_type_label_id` bigint(20) NOT NULL COMMENT 'engine label id', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `engine_type_label_id`) +); +``` + +### Configuration Config Value Table +```sql +CREATE TABLE `linkis_ps_configuration_config_value`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `config_key_id` bigint(20), + `config_value` varchar(500), + `config_label_id`int(20), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `config_label_id`) +); +``` + +### Configuration Category Table +```sql +CREATE TABLE `linkis_ps_configuration_category` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `label_id` int(20) NOT NULL, + `level` int(20) NOT NULL, + `description` varchar(200), + `tag` varchar(200), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_label_id` (`label_id`) +); +``` + +### Tenant Label Config Table +```sql +CREATE TABLE `linkis_cg_tenant_label_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `user` varchar(50) COLLATE utf8_bin NOT NULL, + `creator` varchar(50) COLLATE utf8_bin NOT NULL, + `tenant_value` varchar(128) COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `desc` varchar(100) COLLATE utf8_bin NOT NULL, + `bussiness_user` varchar(50) COLLATE utf8_bin NOT NULL, + `is_valid` varchar(1) COLLATE utf8_bin NOT NULL DEFAULT 'Y' COMMENT 'is valid', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_user_creator` (`user`,`creator`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### User IP Config Table +```sql +CREATE TABLE `linkis_cg_user_ip_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `user` varchar(50) COLLATE utf8_bin NOT NULL, + `creator` varchar(50) COLLATE utf8_bin NOT NULL, + `ip_list` text COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `desc` varchar(100) COLLATE utf8_bin NOT NULL, + `bussiness_user` varchar(50) COLLATE utf8_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_user_creator` (`user`,`creator`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Tenant Department Config Table +```sql +CREATE TABLE `linkis_cg_tenant_department_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT COMMENT 'ID', + `creator` varchar(50) COLLATE utf8_bin NOT NULL COMMENT '应用', + `department` varchar(64) COLLATE utf8_bin NOT NULL COMMENT '部门名称', + `department_id` varchar(16) COLLATE utf8_bin NOT NULL COMMENT '部门ID', + `tenant_value` varchar(128) COLLATE utf8_bin NOT NULL COMMENT '部门租户标签', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + `create_by` varchar(50) COLLATE utf8_bin NOT NULL COMMENT '创建用户', + `is_valid` varchar(1) COLLATE utf8_bin NOT NULL DEFAULT 'Y' COMMENT '是否有效', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_creator_department` (`creator`,`department`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Configuration Template Config Key Table +```sql +CREATE TABLE `linkis_ps_configuration_template_config_key` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `template_name` VARCHAR(200) NOT NULL COMMENT '配置模板名称 冗余存储', + `template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值（预留）', + `validate_range` VARCHAR(50) NULL DEFAULT NULL COMMENT '校验正则(预留) ', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_tid_kid` (`template_uuid`, `key_id`), + UNIQUE INDEX `uniq_tname_kid` (`template_uuid`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Configuration Key Limit For User Table +```sql +CREATE TABLE `linkis_ps_configuration_key_limit_for_user` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `user_name` VARCHAR(50) NOT NULL COMMENT '用户名', + `combined_label_value` VARCHAR(128) NOT NULL COMMENT '组合标签 combined_userCreator_engineType 如 hadoop-IDE,spark-2.4.3', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值（预留）', + `latest_update_template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_com_label_kid` (`combined_label_value`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The Configuration Service provides several RPC methods for configuration management: + +### Configuration RPCs + +#### getGlobalConfiguration +Retrieves global configuration properties: +```java +Map getGlobalConfiguration() +``` + +#### getUserConfiguration +Retrieves user-specific configuration: +```java +Map getUserConfiguration(String user) +``` + +#### updateConfiguration +Updates configuration properties: +```java +void updateConfiguration(String user, Map configurations) +``` + +#### getConfigurationTemplate +Retrieves configuration template for an engine: +```java +ConfigurationTemplate getConfigurationTemplate(String engineType) +``` + +#### validateConfiguration +Validates configuration properties: +```java +ConfigurationValidationResult validateConfiguration(String key, String value) +``` + +#### getEngineConfiguration +Retrieves engine-specific configuration: +```java +Map getEngineConfiguration(String engineType, String version, String user) +``` + +#### updateEngineConfiguration +Updates engine-specific configuration: +```java +void updateEngineConfiguration(String engineType, String version, String user, Map configurations) +``` + +#### listConfigurationKeys +Lists all configuration keys for an engine type: +```java +List listConfigurationKeys(String engineType) +``` + +### Category RPCs + +#### getCategoryConfiguration +Retrieves configuration for a category: +```java +Map getCategoryConfiguration(String category) +``` + +#### updateCategoryConfiguration +Updates configuration for a category: +```java +void updateCategoryConfiguration(String category, Map configurations) +``` + +## Dependencies + +- linkis-mybatis +- linkis-rpc +- linkis-manager-common +- linkis-httpclient +- linkis-label-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ConfigurationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApi.java` +- TenantConfigrationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/TenantConfigrationRestfulApi.java` +- UserIpConfigrationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/UserIpConfigrationRestfulApi.java` +- TemplateRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/TemplateRestfulApi.java` +- AcrossClusterRuleRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/AcrossClusterRuleRestfulApi.java` +- ConfigurationTemplateRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/ConfigurationTemplateRestfulApi.java` + +### MyBatis XML Files +- ConfigMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/ConfigMapper.xml` +- LabelMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/LabelMapper.xml` +- UserTenantMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/UserTenantMapper.xml` +- DepartmentTenantMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/DepartmentTenantMapper.xml` +- UserIpMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/UserIpMapper.xml` +- DepartmentMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/DepartmentMapper.xml` +- AcrossClusterRuleMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/AcrossClusterRuleMapper.xml` +- TemplateConfigKeyMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/TemplateConfigKeyMapper.xml` +- ConfigKeyLimitForUserMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/ConfigKeyLimitForUserMapper.xml` \ No newline at end of file diff --git a/docs/modules/public-enhancements/context.md b/docs/modules/public-enhancements/context.md new file mode 100644 index 0000000000..d1211c0a86 --- /dev/null +++ b/docs/modules/public-enhancements/context.md @@ -0,0 +1,962 @@ +# Context Service + +The Context Service provides context and variable sharing capabilities for the Linkis system. + +## Overview + +This service manages context information and variable sharing across different engines and applications in the Linkis system. + +## Key Components + +### Core Classes +- `LinkisCSApplication` - Main application class +- Context management +- Variable sharing +- Context persistence + +### Features +- Cross-engine context sharing +- Variable management +- Context persistence +- Context versioning + +## API Interfaces + +### Context ID APIs + +#### Create Context ID +``` +POST /api/rest_j/v1/contextservice/createContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/createContextID", + "status": 0, + "message": "success", + "data": { + "contextId": "context-12345" + } +} +``` + +#### Get Context ID +``` +GET /api/rest_j/v1/contextservice/getContextID +``` + +Parameters: +- `contextId`: Context ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getContextID", + "status": 0, + "message": "success", + "data": { + "contextId": { + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Update Context ID +``` +POST /api/rest_j/v1/contextservice/updateContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "updated-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/updateContextID", + "status": 0, + "message": "success" +} +``` + +#### Reset Context ID +``` +POST /api/rest_j/v1/contextservice/resetContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/resetContextID", + "status": 0, + "message": "success" +} +``` + +#### Remove Context ID +``` +POST /api/rest_j/v1/contextservice/removeContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeContextID", + "status": 0, + "message": "success" +} +``` + +#### Search Context ID By Time +``` +GET /api/rest_j/v1/contextservice/searchContextIDByTime +``` + +Parameters: +- `createTimeStart`: Create time start - optional +- `createTimeEnd`: Create time end - optional +- `updateTimeStart`: Update time start - optional +- `updateTimeEnd`: Update time end - optional +- `accessTimeStart`: Access time start - optional +- `accessTimeEnd`: Access time end - optional +- `pageNow`: Page number - optional, default 1 +- `pageSize`: Page size - optional, default 100 + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchContextIDByTime", + "status": 0, + "message": "success", + "data": { + "contextIds": [ + { + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +### Context Value APIs + +#### Get Context Value +``` +POST /api/rest_j/v1/contextservice/getContextValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getContextValue", + "status": 0, + "message": "success", + "data": { + "contextValue": { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } + } +} +``` + +#### Search Context Value +``` +POST /api/rest_j/v1/contextservice/searchContextValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "condition": { + "key": "test-key", + "contextType": "ENV" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchContextValue", + "status": 0, + "message": "success", + "data": { + "contextKeyValue": [ + { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } + ] + } +} +``` + +#### Set Value By Key +``` +POST /api/rest_j/v1/contextservice/setValueByKey +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + }, + "contextValue": { + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/setValueByKey", + "status": 0, + "message": "success" +} +``` + +#### Set Value +``` +POST /api/rest_j/v1/contextservice/setValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyValue": { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/setValue", + "status": 0, + "message": "success" +} +``` + +#### Reset Value +``` +POST /api/rest_j/v1/contextservice/resetValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/resetValue", + "status": 0, + "message": "success" +} +``` + +#### Remove Value +``` +POST /api/rest_j/v1/contextservice/removeValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeValue", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value +``` +POST /api/rest_j/v1/contextservice/removeAllValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValue", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key Prefix And Context Type +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyType": "ENV", + "keyPrefix": "test" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key And Context Type +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyAndContextType +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyType": "ENV", + "contextKey": "test-key" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyAndContextType", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key Prefix +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyPrefix +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "keyPrefix": "test" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyPrefix", + "status": 0, + "message": "success" +} +``` + +#### Clear All Context By ID +``` +POST /api/rest_j/v1/contextservice/clearAllContextByID +``` + +Request Body: +```json +{ + "idList": ["context-12345", "context-67890"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/clearAllContextByID", + "status": 0, + "message": "success", + "data": { + "num": 2 + } +} +``` + +#### Clear All Context By Time +``` +POST /api/rest_j/v1/contextservice/clearAllContextByTime +``` + +Request Body: +```json +{ + "createTimeStart": "2023-01-01 00:00:00", + "createTimeEnd": "2023-12-31 23:59:59", + "updateTimeStart": "2023-01-01 00:00:00", + "updateTimeEnd": "2023-12-31 23:59:59" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/clearAllContextByTime", + "status": 0, + "message": "success", + "data": { + "num": 5 + } +} +``` + +### Context History APIs + +#### Create History +``` +POST /api/rest_j/v1/contextservice/createHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextHistory": { + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/createHistory", + "status": 0, + "message": "success" +} +``` + +#### Remove History +``` +POST /api/rest_j/v1/contextservice/removeHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextHistory": { + "source": "test-source" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeHistory", + "status": 0, + "message": "success" +} +``` + +#### Get Histories +``` +POST /api/rest_j/v1/contextservice/getHistories +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getHistories", + "status": 0, + "message": "success", + "data": { + "contextHistory": [ + { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +#### Get History +``` +POST /api/rest_j/v1/contextservice/getHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getHistory", + "status": 0, + "message": "success", + "data": { + "contextHistory": { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Search History +``` +POST /api/rest_j/v1/contextservice/searchHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "keywords": ["test", "key"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchHistory", + "status": 0, + "message": "success", + "data": { + "contextHistory": [ + { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +### Context Listener APIs + +#### On Bind ID Listener +``` +POST /api/rest_j/v1/contextservice/onBindIDListener +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/onBindIDListener", + "status": 0, + "message": "success", + "data": { + "listener": null + } +} +``` + +#### On Bind Key Listener +``` +POST /api/rest_j/v1/contextservice/onBindKeyListener +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/onBindKeyListener", + "status": 0, + "message": "success", + "data": { + "listener": null + } +} +``` + +#### Heartbeat +``` +POST /api/rest_j/v1/contextservice/heartbeat +``` + +Request Body: +```json +{ + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/heartbeat", + "status": 0, + "message": "success", + "data": { + "ContextKeyValueBean": null + } +} +``` + +## Database Table Structures + +The Context Service manages the following database tables from linkis_ddl.sql: + +### Context Map Table +```sql +CREATE TABLE `linkis_ps_cs_context_map` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(128) DEFAULT NULL, + `context_scope` varchar(32) DEFAULT NULL, + `context_type` varchar(32) DEFAULT NULL, + `props` text, + `value` mediumtext, + `context_id` int(11) DEFAULT NULL, + `keywords` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`), + KEY `idx_keywords` (`keywords`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context Map Listener Table +```sql +CREATE TABLE `linkis_ps_cs_context_map_listener` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `listener_source` varchar(255) DEFAULT NULL, + `key_id` int(11) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context History Table +```sql +CREATE TABLE `linkis_ps_cs_context_history` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `context_id` int(11) DEFAULT NULL, + `source` text, + `context_type` varchar(32) DEFAULT NULL, + `history_json` text, + `keyword` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + KEY `idx_keyword` (`keyword`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context ID Table +```sql +CREATE TABLE `linkis_ps_cs_context_id` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `user` varchar(32) DEFAULT NULL, + `application` varchar(32) DEFAULT NULL, + `source` varchar(255) DEFAULT NULL, + `expire_type` varchar(32) DEFAULT NULL, + `expire_time` datetime DEFAULT NULL, + `instance` varchar(128) DEFAULT NULL, + `backup_instance` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + KEY `idx_instance` (`instance`(128)), + KEY `idx_backup_instance` (`backup_instance`(191)), + KEY `idx_instance_bin` (`instance`(128),`backup_instance`(128)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context Listener Table +```sql +CREATE TABLE `linkis_ps_cs_context_listener` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `listener_source` varchar(255) DEFAULT NULL, + `context_id` int(11) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The Context Service provides several RPC methods for context management: + +### Context RPCs + +#### createContext +Creates a new context: +```java +String createContext(ContextCreationRequest request) +``` + +#### getContext +Retrieves a context: +```java +ContextInfo getContext(String contextId) +``` + +#### deleteContext +Deletes a context: +```java +void deleteContext(String contextId) +``` + +### Variable RPCs + +#### setVariable +Sets a variable in a context: +```java +void setVariable(String contextId, String key, Object value, String valueType) +``` + +#### getVariable +Retrieves a variable from a context: +```java +Object getVariable(String contextId, String key) +``` + +#### removeVariable +Removes a variable from a context: +```java +void removeVariable(String contextId, String key) +``` + +### History RPCs + +#### getContextHistory +Retrieves context history: +```java +List getContextHistory(String contextId) +``` + +#### clearContextHistory +Clears context history: +```java +void clearContextHistory(String contextId) +``` + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ContextRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextRestfulApi.java` +- ContextIDRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextIDRestfulApi.java` +- ContextHistoryRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextHistoryRestfulApi.java` +- ContextListenerRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextListenerRestfulApi.java` + +### MyBatis XML Files +- ContextMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextMapper.xml` +- ContextIDMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextIDMapper.xml` +- ContextHistoryMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextHistoryMapper.xml` +- ContextListenerMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextListenerMapper.xml` + +## Dependencies + +- linkis-cs-server +- linkis-rpc +- linkis-protocol \ No newline at end of file diff --git a/docs/modules/public-enhancements/datasource.md b/docs/modules/public-enhancements/datasource.md new file mode 100644 index 0000000000..6e584cc2cb --- /dev/null +++ b/docs/modules/public-enhancements/datasource.md @@ -0,0 +1,2032 @@ +# DataSource Service + +The DataSource Service provides data source management capabilities for the Linkis system. + +## Overview + +This service manages data source connections, metadata, and provides unified access to various data sources. + +## Key Components + +### Core Classes +- `LinkisDataSourceApplication` - Main application class +- Data source management +- Metadata querying +- Connection testing + +### Features +- Data source registration and management +- Metadata querying +- Connection testing and validation +- Data source versioning +- Access control + +## API Interfaces + +### Data Source Type APIs + +#### Get All Data Source Types +``` +GET /api/rest_j/v1/data-source-manager/type/all +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/type/all", + "status": 0, + "message": "success", + "data": { + "typeList": [ + { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3 + } + ] + } +} +``` + +#### Get Key Definitions By Type +``` +GET /api/rest_j/v1/data-source-manager/key-define/type/{typeId} +``` + +Parameters: +- `typeId`: Data source type ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/key-define/type/{typeId}", + "status": 0, + "message": "success", + "data": { + "keyDefine": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null + } + ] + } +} +``` + +#### Get Key Definitions By Type Name +``` +GET /api/rest_j/v1/data-source-manager/key-define/{typeName} +``` + +Parameters: +- `typeName`: Data source type name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/key-define/{typeName}", + "status": 0, + "message": "success", + "data": { + "keyDefine": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null + } + ] + } +} +``` + +### Data Source Management APIs + +#### Insert Data Source Info (JSON) +``` +POST /api/rest_j/v1/data-source-manager/info/json +``` + +Request Body: +```json +{ + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createSystem": "linkis", + "labels": [ + { + "labelKey": "env", + "labelValue": "production" + } + ], + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/json", + "status": 0, + "message": "success", + "data": { + "insertId": 12345 + } +} +``` + +#### Insert Data Source (JSON Create) +``` +POST /api/rest_j/v1/data-source-manager/info/json/create +``` + +Request Body: +```json +{ + "createUser": "testuser", + "dataSourceTypeName": "starrocks", + "connectParams": { + "host": "localhost", + "port": "9030", + "driverClassName": "com.mysql.jdbc.Driver", + "username": "user", + "password": "password" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/json/create", + "status": 0, + "message": "success", + "data": { + "datasource": { + "id": 12345, + "dataSourceName": "starrocks_testuser_20230101120000", + "dataSourceDesc": null, + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": null, + "parameter": "{\"host\":\"localhost\",\"port\":\"9030\",\"driverClassName\":\"com.mysql.jdbc.Driver\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": null, + "labels": null, + "versionId": null, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Update Data Source Info (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/info/{dataSourceId}/json +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Request Body: +```json +{ + "dataSourceName": "mysql-ds", + "dataSourceDesc": "Updated MySQL Data Source", + "dataSourceTypeId": 1, + "createSystem": "linkis", + "createTime": "1650426189000", + "createUser": "testuser", + "labels": [ + { + "labelKey": "env", + "labelValue": "production" + } + ], + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword", + "database": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/json", + "status": 0, + "message": "success", + "data": { + "updateId": 12345 + } +} +``` + +#### Insert Data Source Parameter (JSON) +``` +POST /api/rest_j/v1/data-source-manager/parameter/{dataSourceId}/json +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Request Body: +```json +{ + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + }, + "comment": "Initial version" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/parameter/{dataSourceId}/json", + "status": 0, + "message": "success", + "data": { + "version": 1 + } +} +``` + +#### Get Data Source Info By ID +``` +GET /api/rest_j/v1/data-source-manager/info/{dataSourceId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Data Source Info By Name +``` +GET /api/rest_j/v1/data-source-manager/info/name/{dataSourceName} +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/name/{dataSourceName}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Published Data Source Info By Name +``` +GET /api/rest_j/v1/data-source-manager/publishedInfo/name/{dataSourceName} +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publishedInfo/name/{dataSourceName}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Published Data Source Info By Type Name, User, IP and Port +``` +GET /api/rest_j/v1/data-source-manager/publishedInfo/{datasourceTypeName}/{datasourceUser}/{ip}/{port} +``` + +Parameters: +- `datasourceTypeName`: Data source type name (required) +- `datasourceUser`: Data source user (required) +- `ip`: IP address (required) +- `port`: Port (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publishedInfo/{datasourceTypeName}/{datasourceUser}/{ip}/{port}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Data Source Info By ID and Version +``` +GET /api/rest_j/v1/data-source-manager/info/{dataSourceId}/{version} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `version`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/{version}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Version List +``` +GET /api/rest_j/v1/data-source-manager/{dataSourceId}/versions +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/versions", + "status": 0, + "message": "success", + "data": { + "versions": [ + { + "versionId": 1, + "dataSourceId": 12345, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "comment": "Initial version", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser" + } + ] + } +} +``` + +#### Publish Data Source By ID +``` +POST /api/rest_j/v1/data-source-manager/publish/{dataSourceId}/{versionId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `versionId`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publish/{dataSourceId}/{versionId}", + "status": 0, + "message": "success" +} +``` + +#### Remove Data Source +``` +DELETE /api/rest_j/v1/data-source-manager/info/delete/{dataSourceId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/delete/{dataSourceId}", + "status": 0, + "message": "success", + "data": { + "removeId": 12345 + } +} +``` + +#### Expire Data Source +``` +PUT /api/rest_j/v1/data-source-manager/info/{dataSourceId}/expire +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/expire", + "status": 0, + "message": "success", + "data": { + "expireId": 12345 + } +} +``` + +#### Get Connect Params By Data Source ID +``` +GET /api/rest_j/v1/data-source-manager/{dataSourceId}/connect-params +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/connect-params", + "status": 0, + "message": "success", + "data": { + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } + } +} +``` + +#### Get Connect Params By Data Source Name +``` +GET /api/rest_j/v1/data-source-manager/name/{dataSourceName}/connect-params +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/name/{dataSourceName}/connect-params", + "status": 0, + "message": "success", + "data": { + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } + } +} +``` + +#### Connect Data Source +``` +PUT /api/rest_j/v1/data-source-manager/{dataSourceId}/{version}/op/connect +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `version`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/{version}/op/connect", + "status": 0, + "message": "success", + "data": { + "ok": true + } +} +``` + +#### Query Data Source By IDs +``` +GET /api/rest_j/v1/data-source-manager/info/ids +``` + +Parameters: +- `ids`: JSON array of data source IDs (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/ids", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + ], + "totalPage": 1 + } +} +``` + +#### Query Data Source +``` +GET /api/rest_j/v1/data-source-manager/info +``` + +Parameters: +- `system`: Create system - optional +- `name`: Data source name - optional +- `typeId`: Data source type ID - optional +- `identifies`: Identifies - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + ], + "totalPage": 1 + } +} +``` + +### Data Source Environment APIs + +#### Insert Data Source Environment (JSON) +``` +POST /api/rest_j/v1/data-source-manager/env/json +``` + +Request Body: +```json +{ + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json", + "status": 0, + "message": "success", + "data": { + "insertId": 12345 + } +} +``` + +#### Insert Data Source Environment Batch (JSON) +``` +POST /api/rest_j/v1/data-source-manager/env/json/batch +``` + +Request Body: +```json +[ + { + "envName": "test-env-1", + "envDesc": "Test Environment 1", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password" + } + }, + { + "envName": "test-env-2", + "envDesc": "Test Environment 2", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3307", + "username": "user", + "password": "password" + } + } +] +``` + +Parameters: +- `system`: System name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json/batch", + "status": 0, + "message": "success", + "data": { + "envs": [ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Test Environment 1", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + }, + { + "id": 12346, + "envName": "test-env-2", + "envDesc": "Test Environment 2", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3307\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Update Data Source Environment Batch (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/env/json/batch +``` + +Request Body: +```json +[ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Updated Test Environment 1", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword" + } + } +] +``` + +Parameters: +- `system`: System name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json/batch", + "status": 0, + "message": "success", + "data": { + "envs": [ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Updated Test Environment 1", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"newpassword\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Get All Environment List By Data Source Type +``` +GET /api/rest_j/v1/data-source-manager/env-list/all/type/{typeId} +``` + +Parameters: +- `typeId`: Data source type ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env-list/all/type/{typeId}", + "status": 0, + "message": "success", + "data": { + "envList": [ + { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Get Environment Entity By ID +``` +GET /api/rest_j/v1/data-source-manager/env/{envId} +``` + +Parameters: +- `envId`: Environment ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}", + "status": 0, + "message": "success", + "data": { + "env": { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + } +} +``` + +#### Remove Environment Entity +``` +DELETE /api/rest_j/v1/data-source-manager/env/{envId} +``` + +Parameters: +- `envId`: Environment ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}", + "status": 0, + "message": "success", + "data": { + "removeId": 12345 + } +} +``` + +#### Update Data Source Environment (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/env/{envId}/json +``` + +Parameters: +- `envId`: Environment ID (required) + +Request Body: +```json +{ + "envName": "test-env", + "envDesc": "Updated Test Environment", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}/json", + "status": 0, + "message": "success", + "data": { + "updateId": 12345 + } +} +``` + +#### Query Data Source Environment +``` +GET /api/rest_j/v1/data-source-manager/env +``` + +Parameters: +- `name`: Environment name - optional +- `typeId`: Data source type ID - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +### Metadata Query APIs + +#### Query Database Info +``` +GET /api/rest_j/v1/datasource/dbs +``` + +Parameters: +- `permission`: Permission filter - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/dbs", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "permission": "READ" + } + ] + } +} +``` + +#### Query Partition Exists +``` +GET /api/rest_j/v1/datasource/partitionExists +``` + +Parameters: +- `database`: Database name (required) +- `table`: Table name (required) +- `partition`: Partition name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/partitionExists", + "status": 0, + "message": "success", + "data": { + "partitionExists": true + } +} +``` + +#### Query Databases With Tables +``` +GET /api/rest_j/v1/datasource/all +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/all", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "tables": [ + { + "name": "test_table" + } + ] + } + ] + } +} +``` + +#### Query Databases With Tables Order By Access Time +``` +GET /api/rest_j/v1/datasource/getByAccessTime +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/getByAccessTime", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "tables": [ + { + "name": "test_table", + "lastAccessTime": "2023-01-01 12:00:00" + } + ] + } + ] + } +} +``` + +#### Query Tables +``` +GET /api/rest_j/v1/datasource/tables +``` + +Parameters: +- `database`: Database name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/tables", + "status": 0, + "message": "success", + "data": { + "tables": [ + { + "name": "test_table" + } + ] + } +} +``` + +#### Query Table Metadata +``` +GET /api/rest_j/v1/datasource/columns +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/columns", + "status": 0, + "message": "success", + "data": { + "columns": [ + { + "name": "id", + "type": "INT", + "comment": "Primary key" + }, + { + "name": "name", + "type": "VARCHAR", + "comment": "Name field" + } + ] + } +} +``` + +#### Get Table Size +``` +GET /api/rest_j/v1/datasource/size +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional +- `partition`: Partition name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/size", + "status": 0, + "message": "success", + "data": { + "sizeInfo": { + "size": "10MB", + "fileCount": 5 + } + } +} +``` + +#### Get Storage Info +``` +GET /api/rest_j/v1/datasource/storage-info +``` + +Parameters: +- `database`: Database name (required) +- `table`: Table name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/storage-info", + "status": 0, + "message": "success", + "data": { + "storageInfo": { + "location": "/path/to/table", + "format": "PARQUET", + "compression": "SNAPPY" + } + } +} +``` + +#### Get Partitions +``` +GET /api/rest_j/v1/datasource/partitions +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/partitions", + "status": 0, + "message": "success", + "data": { + "partitionInfo": [ + { + "name": "dt=20230101", + "location": "/path/to/partition" + } + ] + } +} +``` + +### Data Source Type Management APIs + +#### List Data Source Types +``` +GET /api/rest_j/v1/basedata-manager/datasource-type +``` + +Parameters: +- `searchName`: Search name - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Type +``` +GET /api/rest_j/v1/basedata-manager/datasource-type/{id} +``` + +Parameters: +- `id`: Data source type ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" + } + } +} +``` + +#### Add Data Source Type +``` +POST /api/rest_j/v1/basedata-manager/datasource-type +``` + +Request Body: +```json +{ + "name": "PostgreSQL", + "description": "PostgreSQL Database", + "option": "PostgreSQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "PostgreSQL Database", + "optionEn": "PostgreSQL", + "classifierEn": "Database" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Type +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-type/{id} +``` + +Parameters: +- `id`: Data source type ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Type +``` +PUT /api/rest_j/v1/basedata-manager/datasource-type +``` + +Request Body: +```json +{ + "id": 1, + "name": "MySQL", + "description": "Updated MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "Updated MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Data Source Access Management APIs + +#### List Data Source Accesses +``` +GET /api/rest_j/v1/basedata-manager/datasource-access +``` + +Parameters: +- `searchName`: Search name - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "tokenId": 1, + "serviceId": 1, + "accessTime": "2023-01-01 12:00:00" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Access +``` +GET /api/rest_j/v1/basedata-manager/datasource-access/{id} +``` + +Parameters: +- `id`: Data source access ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "tokenId": 1, + "serviceId": 1, + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Add Data Source Access +``` +POST /api/rest_j/v1/basedata-manager/datasource-access +``` + +Request Body: +```json +{ + "tokenId": 1, + "serviceId": 1 +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Access +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-access/{id} +``` + +Parameters: +- `id`: Data source access ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Access +``` +PUT /api/rest_j/v1/basedata-manager/datasource-access +``` + +Request Body: +```json +{ + "id": 1, + "tokenId": 1, + "serviceId": 2 +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Data Source Type Key Management APIs + +#### List Data Source Type Keys +``` +GET /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Parameters: +- `searchName`: Search name - optional +- `dataSourceTypeId`: Data source type ID - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Type Key +``` +GET /api/rest_j/v1/basedata-manager/datasource-type-key/{id} +``` + +Parameters: +- `id`: Data source type key ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Add Data Source Type Key +``` +POST /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Request Body: +```json +{ + "dataSourceTypeId": 1, + "key": "port", + "name": "Port", + "nameEn": "Port", + "defaultValue": "3306", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Port number", + "descriptionEn": "Port number", + "valueRegex": "" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Type Key +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-type-key/{id} +``` + +Parameters: +- `id`: Data source type key ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Type Key +``` +PUT /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Request Body: +```json +{ + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Updated Host IP", + "descriptionEn": "Updated Host IP", + "valueRegex": "" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +## Database Table Structures + +The DataSource Service uses the following database tables from linkis_ddl.sql: + +### Data Source Table +```sql +CREATE TABLE `linkis_ps_dm_datasource` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `datasource_name` varchar(255) COLLATE utf8_bin NOT NULL, + `datasource_desc` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `datasource_type_id` int(11) NOT NULL, + `create_identify` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `create_system` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `parameter` varchar(2048) COLLATE utf8_bin NULL DEFAULT NULL, + `create_time` datetime NULL DEFAULT CURRENT_TIMESTAMP, + `modify_time` datetime NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `modify_user` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `labels` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `version_id` int(11) DEFAULT NULL COMMENT 'current version id', + `expire` tinyint(1) DEFAULT 0, + `published_version_id` int(11) DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_datasource_name` (`datasource_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Environment Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_env` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `env_name` varchar(32) COLLATE utf8_bin NOT NULL, + `env_desc` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `datasource_type_id` int(11) NOT NULL, + `parameter` varchar(2048) COLLATE utf8_bin DEFAULT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + `modify_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `modify_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_env_name` (`env_name`), + UNIQUE INDEX `uniq_name_dtid` (`env_name`, `datasource_type_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Type Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_type` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `name` varchar(32) COLLATE utf8_bin NOT NULL, + `description` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `option` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `classifier` varchar(32) COLLATE utf8_bin NOT NULL, + `icon` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `layers` int(3) NOT NULL, + `description_en` varchar(255) DEFAULT NULL COMMENT 'english description', + `option_en` varchar(32) DEFAULT NULL COMMENT 'english option', + `classifier_en` varchar(32) DEFAULT NULL COMMENT 'english classifier', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_name` (`name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Type Key Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_type_key` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `data_source_type_id` int(11) NOT NULL, + `key` varchar(32) COLLATE utf8_bin NOT NULL, + `name` varchar(32) COLLATE utf8_bin NOT NULL, + `name_en` varchar(32) COLLATE utf8_bin NULL DEFAULT NULL, + `default_value` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `value_type` varchar(50) COLLATE utf8_bin NOT NULL, + `scope` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `require` tinyint(1) NULL DEFAULT 0, + `description` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `description_en` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `value_regex` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `ref_id` bigint(20) NULL DEFAULT NULL, + `ref_value` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `data_source` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_dstid_key` (`data_source_type_id`, `key`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Version Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_version` +( + `version_id` int(11) NOT NULL AUTO_INCREMENT, + `datasource_id` int(11) NOT NULL, + `parameter` varchar(2048) COLLATE utf8_bin NULL DEFAULT NULL, + `comment` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + `create_time` datetime(0) NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + PRIMARY KEY `uniq_vid_did` (`version_id`, `datasource_id`) USING BTREE +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Access Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_access` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `token_id` int(11) NOT NULL, + `service_id` int(11) NOT NULL, + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The DataSource Service provides several RPC methods for data source management: + +### Data Source RPCs + +#### createDataSource +Creates a new data source: +```java +Long createDataSource(DataSourceCreationRequest request) +``` + +#### getDataSource +Retrieves a data source: +```java +DataSourceInfo getDataSource(Long dataSourceId) +``` + +#### updateDataSource +Updates a data source: +```java +void updateDataSource(DataSourceUpdateRequest request) +``` + +#### deleteDataSource +Deletes a data source: +```java +void deleteDataSource(Long dataSourceId) +``` + +#### listDataSources +Lists data sources with filtering: +```java +List listDataSources(DataSourceQueryRequest request) +``` + +### Metadata RPCs + +#### getMetadata +Retrieves metadata for a data source: +```java +DataSourceMetadata getMetadata(DataSourceMetadataRequest request) +``` + +#### testConnection +Tests connection to a data source: +```java +ConnectionTestResult testConnection(Long dataSourceId) +``` + +#### getTableSchema +Retrieves table schema information: +```java +TableSchema getTableSchema(Long dataSourceId, String database, String table) +``` + +#### getDatabaseList +Retrieves list of databases: +```java +List getDatabaseList(Long dataSourceId) +``` + +#### getTableList +Retrieves list of tables in a database: +```java +List getTableList(Long dataSourceId, String database) +``` + +### Environment RPCs + +#### createEnvironment +Creates a new environment: +```java +Long createEnvironment(EnvironmentCreationRequest request) +``` + +#### getEnvironment +Retrieves an environment: +```java +EnvironmentInfo getEnvironment(Long environmentId) +``` + +#### updateEnvironment +Updates an environment: +```java +void updateEnvironment(EnvironmentUpdateRequest request) +``` + +#### deleteEnvironment +Deletes an environment: +```java +void deleteEnvironment(Long environmentId) +``` + +### Access RPCs + +#### grantAccess +Grants access to a data source: +```java +void grantAccess(DataSourceAccessRequest request) +``` + +#### revokeAccess +Revokes access from a data source: +```java +void revokeAccess(DataSourceAccessRequest request) +``` + +#### checkAccess +Checks if a user has access to a data source: +```java +boolean checkAccess(String user, Long dataSourceId) +``` + +## Dependencies + +- linkis-datasource-manager +- linkis-metadata +- linkis-rpc +- linkis-protocol + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- DataSourceCoreRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java` +- DataSourceAdminRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceAdminRestfulApi.java` +- DataSourceRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/restful/api/DataSourceRestfulApi.java` +- DatasourceTypeRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceTypeRestfulApi.java` +- DatasourceAccessRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceAccessRestfulApi.java` +- DatasourceTypeKeyRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceTypeKeyRestfulApi.java` + +### MyBatis XML Files +- DataSouceMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml` +- DataSourceEnvMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceEnvMapper.xml` +- DataSourceParamKeyMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceParamKeyMapper.xml` +- DataSourceTypeMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceTypeMapper.xml` +- DataSourceVersionMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceVersionMapper.xml` +- DataSourceAccessMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DataSourceAccessMapper.xml` +- DatasourceTypeMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DatasourceTypeMapper.xml` +- DatasourceTypeKeyMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DatasourceTypeKeyMapper.xml` \ No newline at end of file diff --git a/docs/modules/public-enhancements/jobhistory.md b/docs/modules/public-enhancements/jobhistory.md new file mode 100644 index 0000000000..508c55ba92 --- /dev/null +++ b/docs/modules/public-enhancements/jobhistory.md @@ -0,0 +1,579 @@ +# JobHistory Service + +The JobHistory service manages job execution history and provides querying capabilities for completed tasks in the Linkis system. + +## Overview + +This service tracks and stores information about job executions, including task status, execution results, logs, and performance metrics. It provides APIs for querying job history, statistics, and diagnostics. + +## Key Components + +### Core Classes +- `LinkisJobHistoryApplication` - Main application class +- Job history persistence and querying +- Task statistics and metrics collection +- Job diagnosis and failure analysis + +### Features +- Job execution history tracking +- Task result and log storage +- Performance statistics and metrics +- Job failure diagnosis +- Historical data querying and filtering + +## API Interfaces + +### Query Task by ID +``` +GET /api/rest_j/v1/jobhistory/{id}/get +``` + +Parameters: +- `id` (required): Task ID +- `brief` (optional): If true, only returns brief info + +Response: +```json +{ + "method": "/api/jobhistory/{id}/get", + "status": 0, + "message": "success", + "data": { + "task": { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql", + "params": { + "configuration": { + "runtime": { + "spark.executor.instances": "2" + } + } + } + } + } +} +``` + +### List Tasks +``` +GET /api/rest_j/v1/jobhistory/list +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `taskID` (optional): Specific task ID +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `isAdminView` (optional): Admin view flag +- `isDeptView` (optional): Department view flag +- `instance` (optional): Instance filter +- `engineInstance` (optional): Engine instance filter +- `runType` (optional): Run type filter + +Response: +```json +{ + "method": "/api/jobhistory/list", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ], + "totalPage": 100 + } +} +``` + +### List Undone Tasks +``` +GET /api/rest_j/v1/jobhistory/listundonetasks +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter (default: "Running,Inited,Scheduled") +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `startTaskID` (optional): Start task ID +- `engineType` (optional): Engine type filter +- `creator` (optional): Creator filter + +Response: +```json +{ + "method": "/api/jobhistory/listundonetasks", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Running", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "0.5", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ], + "totalPage": 10 + } +} +``` + +### List Undone Task Count +``` +GET /api/rest_j/v1/jobhistory/listundone +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `startTaskID` (optional): Start task ID +- `engineType` (optional): Engine type filter +- `creator` (optional): Creator filter + +Response: +```json +{ + "method": "/api/jobhistory/listundone", + "status": 0, + "message": "success", + "data": { + "totalPage": 10 + } +} +``` + +### List Tasks by Task IDs +``` +GET /api/rest_j/v1/jobhistory/list-taskids +``` + +Parameters: +- `taskID` (required): Comma-separated list of task IDs (max 30) + +Response: +```json +{ + "method": "/api/jobhistory/list-taskids", + "status": 0, + "message": "success", + "data": { + "jobHistoryList": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ] + } +} +``` + +### Get Job Extra Info +``` +GET /api/rest_j/v1/jobhistory/job-extra-info +``` + +Parameters: +- `jobId` (required): Job ID + +Response: +```json +{ + "method": "/api/jobhistory/job-extra-info", + "status": 0, + "message": "success", + "data": { + "metricsMap": { + "executionCode": "SELECT * FROM table", + "runtime": "300s", + // Additional metrics data + } + } +} +``` + +### Download Job List +``` +GET /api/rest_j/v1/jobhistory/download-job-list +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `taskID` (optional): Specific task ID +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `isAdminView` (optional): Admin view flag +- `isDeptView` (optional): Department view flag +- `instance` (optional): Instance filter +- `engineInstance` (optional): Engine instance filter + +Response: +``` +Excel file download +``` + +### List Duration Top Tasks +``` +GET /api/rest_j/v1/jobhistory/listDurationTop +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) + +Response: +```json +{ + "method": "/api/jobhistory/listDurationTop", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ] + } +} +``` + +### Query Failed Task Diagnosis +``` +GET /api/rest_j/v1/jobhistory/diagnosis-query +``` + +Parameters: +- `taskID` (required): Task ID + +Response: +```json +{ + "method": "/api/jobhistory/diagnosis-query", + "status": 0, + "message": "success", + "data": { + "diagnosisMsg": "Diagnosis message content" + } +} +``` + +### Get Governance Station Admin Info +``` +GET /api/rest_j/v1/jobhistory/governanceStationAdmin +``` + +Response: +```json +{ + "method": "/api/jobhistory/governanceStationAdmin", + "status": 0, + "message": "success", + "data": { + "admin": true, + "historyAdmin": true, + "deptAdmin": false, + "canResultSet": true, + "errorMsgTip": "Error message tip" + } +} +``` + +### Task Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/taskCount +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/taskCount", + "status": 0, + "message": "success", + "data": { + "sumCount": 100, + "succeedCount": 80, + "failedCount": 15, + "cancelledCount": 5 + } +} +``` + +### Engine Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/engineCount +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/engineCount", + "status": 0, + "message": "success", + "data": { + "countEngine": 50, + "countEngineSucceed": 40, + "countEngineFailed": 8, + "countEngineShutting": 2 + } +} +``` + +### Add Observe Info +``` +POST /api/rest_j/v1/jobhistory/setting/addObserveInfo +``` + +Request Body: +```json +{ + "taskId": 12345, + "receiver": "user@example.com", + "extra": { + "title": "Job Alert", + "detail": "Job execution details" + }, + "monitorLevel": "HIGH", + "subSystemId": "subsystem1" +} +``` + +Response: +```json +{ + "method": "/api/jobhistory/setting/addObserveInfo", + "status": 0, + "message": "success" +} +``` + +### Delete Observe Info +``` +GET /api/rest_j/v1/jobhistory/setting/deleteObserveInfo +``` + +Parameters: +- `taskId` (required): Task ID + +Response: +```json +{ + "method": "/api/jobhistory/setting/deleteObserveInfo", + "status": 0, + "message": "success" +} +``` + +## Database Table Structures + +The JobHistory service uses the following database tables for job execution history management: + +### Job History Group History Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The JobHistory service provides several RPC methods for job history management: + +### Job History Query RPCs + +#### getJobHistoryById +Retrieves job history by ID: +```java +JobHistory getJobHistoryById(Long jobId) +``` + +#### searchJobHistory +Searches job history with filters: +```java +List searchJobHistory(JobHistorySearchRequest request) +``` + +#### updateJobHistory +Updates job history information: +```java +void updateJobHistory(JobHistory jobHistory) +``` + +#### deleteJobHistory +Deletes job history: +```java +void deleteJobHistory(Long jobId) +``` + +### Job Statistics RPCs + +#### taskExecutionStatistics +Retrieves task execution statistics: +```java +JobStatistics taskExecutionStatistics(StatisticsRequest request) +``` + +#### engineExecutionStatistics +Retrieves engine execution statistics: +```java +JobStatistics engineExecutionStatistics(StatisticsRequest request) +``` + +### Job Diagnosis RPCs + +#### diagnoseJob +Performs job diagnosis: +```java +JobDiagnosis diagnoseJob(Long jobId) +``` + +#### getDiagnosisInfo +Retrieves diagnosis information: +```java +JobDiagnosis getDiagnosisInfo(Long jobId) +``` + +## Dependencies + +- linkis-jobhistory-server +- linkis-rpc +- linkis-protocol +- linkis-commons +- Database drivers (MySQL, etc.) \ No newline at end of file diff --git a/docs/modules/public-enhancements/publicservice.md b/docs/modules/public-enhancements/publicservice.md new file mode 100644 index 0000000000..6606d8a296 --- /dev/null +++ b/docs/modules/public-enhancements/publicservice.md @@ -0,0 +1,151 @@ +# Public Service + +The Public Service provides core public services for the Linkis system. + +## Overview + +This service provides common public services including file system operations, variable management, and other shared functionalities. + +## Key Components + +### Core Classes +- `LinkisPublicServiceApp` - Main application class +- File system operations +- Variable management +- Shared service utilities + +### Features +- File system operations (upload, download, list) +- Variable management +- Shared service utilities +- Common REST APIs + +## API Interfaces + +### File System Operations +``` +POST /api/rest_j/v1/filesystem/upload +``` + +Request: +``` +multipart/form-data with file content +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/filesystem/upload", + "status": 0, + "message": "success", + "data": { + "path": "/path/to/uploaded/file" + } +} +``` + +### Variable Management +``` +POST /api/rest_j/v1/variable/add +``` + +Request Body: +```json +{ + "key": "variableKey", + "value": "variableValue", + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/variable/add", + "status": 0, + "message": "success", + "data": {} +} +``` + +## Database Table Structures + +The Public Service manages the following database tables: + +### File System Metadata Table +```sql +CREATE TABLE linkis_filesystem_meta ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + user_name VARCHAR(32) NOT NULL, + path VARCHAR(500) NOT NULL, + file_type VARCHAR(50), + file_size BIGINT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + UNIQUE KEY uk_user_path (user_name, path) +); +``` + +### Variable Table +```sql +CREATE TABLE linkis_variable ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + user_name VARCHAR(32) NOT NULL, + key_name VARCHAR(128) NOT NULL, + value TEXT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + UNIQUE KEY uk_user_key (user_name, key_name) +); +``` + +## RPC Methods + +The Public Service provides several RPC methods for common operations: + +### File System RPCs + +#### uploadFile +Uploads a file: +```java +String uploadFile(FileUploadRequest request) +``` + +#### downloadFile +Downloads a file: +```java +FileContent downloadFile(String path, String user) +``` + +#### listFiles +Lists files in a directory: +```java +List listFiles(String path, String user) +``` + +### Variable RPCs + +#### setVariable +Sets a variable: +```java +void setVariable(String key, String value, String user) +``` + +#### getVariable +Retrieves a variable: +```java +String getVariable(String key, String user) +``` + +#### deleteVariable +Deletes a variable: +```java +void deleteVariable(String key, String user) +``` + +## Dependencies + +- linkis-filesystem +- linkis-variable +- linkis-rpc +- linkis-protocol \ No newline at end of file From 6b9f3221fefeb3b7fd182e69d1d9ee32c3bfa18a Mon Sep 17 00:00:00 2001 From: Kazuto Iris <78157415+kazutoiris@users.noreply.github.com> Date: Thu, 30 Oct 2025 08:15:06 -0400 Subject: [PATCH 2/2] docs: move monitor service documentation The `monitor.md` file has been moved from the `microservice-governance` module to the `public-enhancements` module to better reflect its scope and usage. --- .../{microservice-governance => public-enhancements}/monitor.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/modules/{microservice-governance => public-enhancements}/monitor.md (100%) diff --git a/docs/modules/microservice-governance/monitor.md b/docs/modules/public-enhancements/monitor.md similarity index 100% rename from docs/modules/microservice-governance/monitor.md rename to docs/modules/public-enhancements/monitor.md