[Source-Github] - Rate Limiter check should support Enterprise self hosted API endpoints

[kkr2320]

Connector Name

source-github

Connector Version

1.8.26

What step the error happened?

Configuring a new connector

Relevant information

In the utils.py MultipleTokenAuthenticatorWithRateLimiter , within _check_token_limits function , the github API endpoint is harded code to https://api.github.com . This prevents using the connector for enterprise self hosted github environments. The API url is passed part of the connector config is not used. This function should be modified to use the api_url provided.

Relevant log output

2025-02-26 20:33:19 - INFO - 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/opt/airflow/pyairbyte-sharepoint/bin/source-github", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/run.py", line 17, in run
    launch(source, sys.argv[1:])
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 341, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 163, in run
    yield from map(
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py", line 230, in discover
    catalog = self.source.discover(self.logger, config)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py", line 89, in discover
    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/source.py", line 212, in streams
    authenticator = self._get_authenticator(config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/source.py", line 134, in _get_authenticator
    return MultipleTokenAuthenticatorWithRateLimiter(tokens=tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/utils.py", line 64, in __init__
    self.check_all_tokens()
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/utils.py", line 139, in check_all_tokens
    self._check_token_limits(token)
  File "/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/utils.py", line 125, in _check_token_limits
    remaining_info_core = rate_limit_info.get("core")
                          ^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
2025-02-26 20:33:19 - ERROR - Something went wrong in the connector. See the logs for more details.
2025-02-26 20:33:19 - INFO - {"type":"TRACE","trace":{"type":"ERROR","emitted_at":1740601999402,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"'NoneType' object has no attribute 'get'","stack_trace":"Traceback (most recent call last):\n  File \"/opt/airflow/pyairbyte-sharepoint/bin/source-github\", line 8, in <module>\n    sys.exit(run())\n             ^^^^^\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/run.py\", line 17, in run\n    launch(source, sys.argv[1:])\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py\", line 341, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py\", line 163, in run\n    yield from map(\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py\", line 230, in discover\n    catalog = self.source.discover(self.logger, config)\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/airbyte_cdk/sources/abstract_source.py\", line 89, in discover\n    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]\n                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/source.py\", line 212, in streams\n    authenticator = self._get_authenticator(config)\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/source.py\", line 134, in _get_authenticator\n    return MultipleTokenAuthenticatorWithRateLimiter(tokens=tokens)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/utils.py\", line 64, in __init__\n    self.check_all_tokens()\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/utils.py\", line 139, in check_all_tokens\n    self._check_token_limits(token)\n  File \"/opt/airflow/pyairbyte-sharepoint/lib/python3.11/site-packages/source_github/utils.py\", line 125, in _check_token_limits\n    remaining_info_core = rate_limit_info.get(\"core\")\n                          ^^^^^^^^^^^^^^^^^^^\nAttributeError: 'NoneType' object has no attribute 'get'\n","failure_type":"system_error"}}}

Contribute

• Yes, I want to contribute