LEAN CLI CharlesSchwab Brokerage support (#517)

* feat: CharlesSchwab in Readme

* remove: obsolete TDAmeritrade

* feat: add project_id to authorize in auth0_client

* feat: use default project id like 0 if configuration is none

* Revert "feat: use default project id like 0 if configuration is none"

This reverts commit ebaa82b.

* Revert "feat: add project_id to authorize in auth0_client"

This reverts commit cf0d691.

* feat: add project_id to authorize in auth0_client

* refactor: set project_id in config_manager

* remove: not used imports

* refactor: use negative local id if cloud id is not provided

* rename: get project id method
refactor: return auth url with any project_id

* refactor: get project id
refactor: Readme

* refactor: Readme

* feat: set project id always in default configs

* feat: validate config value on empty

* refactor: description of project-id in Readme

* feat: new 'require_project_id' property in AuthConfiguration
feat: prompt user to write project id if it required
feat: use project_id always -1
refactor: remove extra project-id from CharlesSchwab parameters in Readme

* refactor: carry out prompt import into get_project_id

* refactor: remove extra project id from Readme

* refactor: use int type implicitly

* fix: underscore in "require-project-id" param

Author: Romazes

README.md

@@ -150,7 +150,7 @@ Options:
   -d, --detach                    Run the backtest in a detached Docker container and return immediately
   --debug [pycharm|ptvsd|debugpy|vsdbg|rider|local-platform]
                                   Enable a certain debugging method (see --help for more information)
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|CharlesSchwab|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -178,6 +178,8 @@ Options:
   --kraken-api-secret TEXT        Your Kraken API secret
   --kraken-verification-tier [Starter|Intermediate|Pro]
                                   Your Kraken Verification Tier
+  --charles-schwab-account-number TEXT
+                                  The CharlesSchwab account number
   --iqfeed-iqconnect TEXT         The path to the IQConnect binary
   --iqfeed-username TEXT          Your IQFeed username
   --iqfeed-password TEXT          Your IQFeed password
@@ -349,9 +351,9 @@ Usage: lean cloud live deploy [OPTIONS] PROJECT
   --notify-insights.
 
 Options:
-  --brokerage [Paper Trading|Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|TDAmeritrade|Bybit|TradeStation|Alpaca]
+  --brokerage [Paper Trading|Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|CharlesSchwab|Bybit|TradeStation|Alpaca]
                                   The brokerage to use
-  --data-provider-live [QuantConnect|Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|TDAmeritrade|Polygon|IEX|CoinApi|Bybit|TradeStation|Alpaca]
+  --data-provider-live [QuantConnect|Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|CharlesSchwab|Polygon|IEX|CoinApi|Bybit|TradeStation|Alpaca]
                                   The live data provider to use
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -427,11 +429,8 @@ Options:
   --kraken-api-secret TEXT        Your Kraken API secret
   --kraken-verification-tier [Starter|Intermediate|Pro]
                                   Your Kraken Verification Tier
-  --tdameritrade-api-key TEXT     Your TDAmeritrade API key
-  --tdameritrade-access-token TEXT
-                                  Your TDAmeritrade OAuth Access Token
-  --tdameritrade-account-number TEXT
-                                  Your TDAmeritrade account number
+  --charles-schwab-account-number TEXT
+                                  The CharlesSchwab account number
   --bybit-api-key TEXT            Your Bybit API key
   --bybit-api-secret TEXT         Your Bybit API secret
   --bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
@@ -852,7 +851,7 @@ Usage: lean data download [OPTIONS]
   https://www.quantconnect.com/datasets
 
 Options:
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|CharlesSchwab|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   The name of the downloader data provider.
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -880,6 +879,8 @@ Options:
   --kraken-api-secret TEXT        Your Kraken API secret
   --kraken-verification-tier [Starter|Intermediate|Pro]
                                   Your Kraken Verification Tier
+  --charles-schwab-account-number TEXT
+                                  The CharlesSchwab account number
   --iqfeed-iqconnect TEXT         The path to the IQConnect binary
   --iqfeed-username TEXT          Your IQFeed username
   --iqfeed-password TEXT          Your IQFeed password
@@ -1289,11 +1290,11 @@ Options:
   --environment TEXT              The environment to use
   --output DIRECTORY              Directory to store results in (defaults to PROJECT/live/TIMESTAMP)
   -d, --detach                    Run the live deployment in a detached Docker container and return immediately
-  --brokerage [Paper Trading|Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|TDAmeritrade|Bybit|TradeStation|Alpaca]
+  --brokerage [Paper Trading|Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|CharlesSchwab|Bybit|TradeStation|Alpaca]
                                   The brokerage to use
-  --data-provider-live [Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|TDAmeritrade|IQFeed|Polygon|IEX|CoinApi|ThetaData|Custom data only|Bybit|TradeStation|Alpaca]
+  --data-provider-live [Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|CharlesSchwab|IQFeed|Polygon|IEX|CoinApi|ThetaData|Custom data only|Bybit|TradeStation|Alpaca]
                                   The live data provider to use
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Bybit|TradeStation|Alpaca]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|CharlesSchwab|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -1382,11 +1383,8 @@ Options:
   --kraken-api-secret TEXT        Your Kraken API secret
   --kraken-verification-tier [Starter|Intermediate|Pro]
                                   Your Kraken Verification Tier
-  --tdameritrade-api-key TEXT     Your TDAmeritrade API key
-  --tdameritrade-access-token TEXT
-                                  Your TDAmeritrade OAuth Access Token
-  --tdameritrade-account-number TEXT
-                                  Your TDAmeritrade account number
+  --charles-schwab-account-number TEXT
+                                  The CharlesSchwab account number
   --bybit-api-key TEXT            Your Bybit API key
   --bybit-api-secret TEXT         Your Bybit API secret
   --bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
@@ -1730,7 +1728,7 @@ Options:
   --parameter <TEXT FLOAT FLOAT FLOAT>...
                                   The 'parameter min max step' pairs configuring the parameters to optimize
   --constraint TEXT               The 'statistic operator value' pairs configuring the constraints of the optimization
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|CharlesSchwab|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --download-data                 Update the Lean configuration file to download data from the QuantConnect API, alias
                                   for --data-provider-historical QuantConnect
@@ -1771,6 +1769,8 @@ Options:
   --kraken-api-secret TEXT        Your Kraken API secret
   --kraken-verification-tier [Starter|Intermediate|Pro]
                                   Your Kraken Verification Tier
+  --charles-schwab-account-number TEXT
+                                  The CharlesSchwab account number
   --iqfeed-iqconnect TEXT         The path to the IQConnect binary
   --iqfeed-username TEXT          Your IQFeed username
   --iqfeed-password TEXT          Your IQFeed password
@@ -1989,7 +1989,7 @@ Usage: lean research [OPTIONS] PROJECT
 
 Options:
   --port INTEGER                  The port to run Jupyter Lab on (defaults to 8888)
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|CharlesSchwab|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -2017,6 +2017,8 @@ Options:
   --kraken-api-secret TEXT        Your Kraken API secret
   --kraken-verification-tier [Starter|Intermediate|Pro]
                                   Your Kraken Verification Tier
+  --charles-schwab-account-number TEXT
+                                  The CharlesSchwab account number
   --iqfeed-iqconnect TEXT         The path to the IQConnect binary
   --iqfeed-username TEXT          Your IQFeed username
   --iqfeed-password TEXT          Your IQFeed password

lean/components/api/auth0_client.py

@@ -52,15 +52,17 @@ def read(self, brokerage_id: str) -> QCAuth0Authorization:
             return QCAuth0Authorization(authorization=None)
 
     @staticmethod
-    def authorize(brokerage_id: str, logger: Logger) -> None:
+    def authorize(brokerage_id: str, logger: Logger,  project_id: int) -> None:
         """Starts the authorization process for a brokerage.
 
         :param brokerage_id: the id of the brokerage to start the authorization process for
         :param logger: the logger instance to use
+        :param project_id: The local or cloud project_id
         """
         from webbrowser import open
 
-        full_url = f"{API_BASE_URL}live/auth0/authorize?brokerage={brokerage_id}"
+        full_url = f"{API_BASE_URL}live/auth0/authorize?brokerage={brokerage_id}&projectId={project_id}"
+
         logger.info(f"Please open the following URL in your browser to authorize the LEAN CLI.")
         logger.info(full_url)
         open(full_url)

lean/components/config/lean_config_manager.py

@@ -245,6 +245,8 @@ def get_complete_lean_config(self,
             "job-user-id": self._cli_config_manager.user_id.get_value(default="0"),
             "api-access-token": self._cli_config_manager.api_token.get_value(default=""),
             "job-organization-id": get_organization(config),
+            "project-id": self._project_config_manager.get_project_id_from_project_config(
+                algorithm_file.parent if algorithm_file else None),
 
             "ib-host": "127.0.0.1",
             "ib-port": "4002",

lean/components/config/project_config_manager.py

@@ -68,6 +68,34 @@ def get_local_id(self, project_directory: Path) -> int:
 
         return project_id
 
+    def get_project_id_from_project_config(self, project_directory: Path) -> int:
+        """
+        Resolves the project ID from the configuration.
+
+        Args:
+            project_directory (Path): The directory of the project. If None,
+                                      it indicates the directory is unavailable.
+
+        Returns:
+            int: Returns the 'cloud-id' if available.
+                 If 'cloud-id' is missing, returns the negative of 'local-id'.
+                 If neither is found nor if project_directory is None, returns -1.
+        """
+        if project_directory is None:
+            return -1
+
+        project_config = self.get_project_config(project_directory)
+
+        cloud_id = project_config.get("cloud-id")
+        if cloud_id is not None:
+            return cloud_id
+
+        local_id = project_config.get("local-id")
+        if local_id is not None:
+            return -local_id  # Local ID must be negative.
+
+        return -1  # Return -1 if no valid IDs are found
+
     def get_latest_live_directory(self, project_directory: Path) -> Path:
         """Returns the path of the latest live directory.
 

lean/components/util/auth0_helper.py

@@ -16,12 +16,13 @@
 from lean.components.util.logger import Logger
 
 
-def get_authorization(auth0_client: Auth0Client, brokerage_id: str, logger: Logger) -> QCAuth0Authorization:
+def get_authorization(auth0_client: Auth0Client, brokerage_id: str, logger: Logger, project_id: int) -> QCAuth0Authorization:
     """Gets the authorization data for a brokerage, authorizing if necessary.
 
     :param auth0_client: An instance of Auth0Client, containing methods to interact with live/auth0/* API endpoints.
     :param brokerage_id: The ID of the brokerage to get the authorization data for.
     :param logger: An instance of Logger, handling all output printing.
+    :param project_id: The local or cloud project_id.
     :return: The authorization data for the specified brokerage.
     """
     from time import time, sleep
@@ -31,7 +32,7 @@ def get_authorization(auth0_client: Auth0Client, brokerage_id: str, logger: Logg
         return data
 
     start_time = time()
-    auth0_client.authorize(brokerage_id, logger)
+    auth0_client.authorize(brokerage_id, logger, project_id)
 
     # keep checking for new data every 5 seconds for 7 minutes
     while time() - start_time < 420:

lean/models/configuration.py

@@ -400,6 +400,7 @@ class AuthConfiguration(InternalInputUserInput):
 
     def __init__(self, config_json_object):
         super().__init__(config_json_object)
+        self.require_project_id = config_json_object.get("require-project-id", False)
 
     def factory(config_json_object) -> 'AuthConfiguration':
         """Creates an instance of the child classes.

lean/models/json_module.py

@@ -175,6 +175,20 @@ def convert_variable_to_lean_key(self, variable_key: str) -> str:
         """
         return variable_key.replace('_', '-')
 
+    def get_project_id(self, default_project_id: int, require_project_id: bool) -> int:
+        """Retrieve the project ID, prompting the user if required and default is invalid.
+
+        :param default_project_id: The default project ID to use.
+        :param require_project_id: Flag to determine if prompting is necessary.
+        :return: A valid project ID.
+        """
+        from click import prompt
+        project_id: int = default_project_id
+        if require_project_id and project_id <= 0:
+            project_id = prompt("Please enter any cloud project ID to proceed with Auth0 authentication",
+                                -1, show_default=False)
+        return project_id
+
     def config_build(self,
                      lean_config: Dict[str, Any],
                      logger: Logger,
@@ -219,7 +233,11 @@ def config_build(self,
                 logger.debug(f"skipping configuration '{configuration._id}': no choices available.")
                 continue
             elif isinstance(configuration, AuthConfiguration):
-                auth_authorizations = get_authorization(container.api_client.auth0, self._display_name.lower(), logger)
+                lean_config["project-id"] = self.get_project_id(lean_config["project-id"],
+                                                                configuration.require_project_id)
+                logger.debug(f'project_id: {lean_config["project-id"]}')
+                auth_authorizations = get_authorization(container.api_client.auth0, self._display_name.lower(),
+                                                        logger, lean_config["project-id"])
                 logger.debug(f'auth: {auth_authorizations}')
                 configuration._value = auth_authorizations.get_authorization_config_without_account()
                 for inner_config in self._lean_configs: