Skip to content

[DSIP-83][datasource-plugin] Add DolphinDB datasource #16962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: dev
Choose a base branch
from

Conversation

eco8848
Copy link

@eco8848 eco8848 commented Jan 15, 2025

Purpose of the pull request

  • This pull request adds DolphinDB datasource.

close #16818

Brief change log

  • Add DolphinDB datasource module.
  • Add DolphinDB documentation.
  • Add DolphinDB constant definitions, connection parameters, data source processors and other components.
  • Add DolphinDB JDBC driver dependency.

Verify this pull request

This change added tests and can be verified as follows:

  • Add DataSourceUtilsTest class to test the datasource utility methods.
  • Add DolphinDBDataSourceChannelFactoryTest class to verify the creation methods of the datasource channel factory.
  • Add DolphinDBDataSourceChannelTest class to validate the creation of the datasource client through the channel.
  • Add DolphinDBDataSourceProcessorTest class to test various functionalities of the datasource processor.

Copy link

boring-cyborg bot commented Jan 15, 2025

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

@eco8848
Copy link
Author

eco8848 commented Jan 15, 2025

Related DSIP #16818

@SbloodyS SbloodyS changed the title [Feature-15953][datasource-plugin] Add DolphinDB datasource [DSIP-83][datasource-plugin] Add DolphinDB datasource Jan 15, 2025
@SbloodyS SbloodyS added this to the 3.3.0 milestone Jan 15, 2025
@SbloodyS SbloodyS added the DSIP label Jan 15, 2025

String[] hostSeperator = connectionParams.getAddress().split(Constants.DOUBLE_SLASH);
String[] hostPortArray = hostSeperator[hostSeperator.length - 1].split(Constants.COMMA);
dataSourceParamDTO.setPort(Integer.parseInt(hostPortArray[0].split(Constants.COLON)[1]));

Check notice

Code scanning / CodeQL

Missing catch of NumberFormatException Note

Potential uncaught 'java.lang.NumberFormatException'.
Copy link
Member

@SbloodyS SbloodyS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add dolphinscheduler-datasource-dolphindb to https://github.com/apache/dolphinscheduler/blob/dev/config/plugins_config

@davidzollo davidzollo added the first time contributor First-time contributor label Jan 22, 2025
@SbloodyS
Copy link
Member

SbloodyS commented Feb 4, 2025

Please fix CI errors. @eco8848

@eco8848
Copy link
Author

eco8848 commented Feb 11, 2025

I have referred to the official license requirements at https://dolphinscheduler.apache.org/zh-cn/docs/3.2.2/contribute/join/DS-License. It is necessary to add a description of the newly introduced third-party library in tools/dependencies/known-dependencies.txt, but it seems to conflict with #16838. What should I do now to comply with the requirements? @SbloodyS

@eco8848
Copy link
Author

eco8848 commented Feb 13, 2025

The CI errors have already been fixed. @SbloodyS

@eco8848 eco8848 requested a review from SbloodyS February 13, 2025 02:03
@SbloodyS SbloodyS removed this from the 3.3.0 milestone Feb 23, 2025
@eco8848
Copy link
Author

eco8848 commented Feb 28, 2025

I saw that there is a new check requires a milestone label. So is there a general plan for the integration of this PR? @SbloodyS

@davidzollo davidzollo added 3.3.0 and removed 3.3.0 labels Mar 17, 2025
@davidzollo davidzollo added this to the 3.3.0 milestone Mar 17, 2025
Copy link

@SbloodyS SbloodyS modified the milestones: 3.3.0-alpha, 3.3.1 Mar 30, 2025
@nielifeng nielifeng requested a review from Copilot April 21, 2025 07:09
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request introduces a new DolphinDB datasource plugin to support DolphinDB connections in the system. Key changes include:

  • Implementation of the DolphinDB datasource module with connection parameters, processors, and client/channel classes.
  • Addition of comprehensive test cases for various components of the datasource.
  • Updates to documentation and dependency management for the new JDBC driver.

Reviewed Changes

Copilot reviewed 23 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/utils/DataSourceUtilsTest.java Added tests for datasource utilities.
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceProcessorTest.java Added tests for datasource processor functionality.
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannelTest.java Added tests for datasource channel client creation.
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannelFactoryTest.java Added tests for datasource channel factory.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceProcessor.java Implements the datasource processor for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceParamDTO.java Defines the parameter DTO for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBConnectionParam.java Implements connection parameters for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBPooledDataSourceClient.java Defines the pooled datasource client for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannelFactory.java Implements the datasource channel factory for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannel.java Defines the datasource channel for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBAdHocDataSourceClient.java Defines the ad hoc datasource client for DolphinDB.
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/constants/DataSourceConstants.java Adds constants for the DolphinDB driver, JDBC URL prefix, and validation query.
docs/docs/zh/guide/datasource/dolphindb.md & docs/docs/en/guide/datasource/dolphindb.md Added documentation for the new DolphinDB datasource.
docs/configs/docsdev.js Updated navigation configuration to include DolphinDB documentation.
Files not reviewed (5)
  • config/plugins_config: Language not supported
  • dolphinscheduler-bom/pom.xml: Language not supported
  • dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-all/pom.xml: Language not supported
  • dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-dolphindb/pom.xml: Language not supported
  • dolphinscheduler-datasource-plugin/pom.xml: Language not supported

Comment on lines +46 to +47
if (!baseDataSourceParamDTO.getType().equals(DbType.REDSHIFT)) {
// due to redshift use not regular hosts
Copy link
Preview

Copilot AI Apr 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The condition incorrectly checks for DbType.REDSHIFT instead of DbType.DOLPHINDB, which may lead to incorrect host validation. Adjust the condition to compare against DbType.DOLPHINDB.

Suggested change
if (!baseDataSourceParamDTO.getType().equals(DbType.REDSHIFT)) {
// due to redshift use not regular hosts
if (baseDataSourceParamDTO.getType().equals(DbType.DOLPHINDB)) {
// validate host for DolphinDB

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend document DSIP first time contributor First-time contributor test UI ui and front end related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[DSIP-83][Feature] Add DolphinDB datasource support to DolphinScheduler
3 participants