feat: support metadata output from LLM #274

kdziedzic68 · 2025-01-15T09:39:32Z

No description provided.

packages/ragbits-core/src/ragbits/core/llms/base.py

github-actions · 2025-01-15T11:00:23Z

Code Coverage Summary

Filename                                                                                                      Stmts    Miss  Cover    Missing
----------------------------------------------------------------------------------------------------------  -------  ------  -------  --------------------------------------------
packages/__init__.py                                                                                              0       0  100.00%
packages/ragbits-cli/src/ragbits/cli/__init__.py                                                                 31       4  87.10%   73-74, 81-82
packages/ragbits-cli/src/ragbits/cli/_utils.py                                                                   23       4  82.61%   45, 62-64
packages/ragbits-cli/src/ragbits/cli/state.py                                                                    55       8  85.45%   48-49, 58, 61-62, 105, 112-113
packages/ragbits-conversations/src/ragbits/conversations/__init__.py                                              0       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/__init__.py                                      0       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/__init__.py                          3       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/base.py                             10       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/llm.py                              29       1  96.55%   79
packages/ragbits-conversations/tests/unit/history/test_llm_compressor.py                                         64       0  100.00%
packages/ragbits-core/src/ragbits/core/__init__.py                                                                6       2  66.67%   8-9
packages/ragbits-core/src/ragbits/core/cli.py                                                                     6       0  100.00%
packages/ragbits-core/src/ragbits/core/config.py                                                                 17       0  100.00%
packages/ragbits-core/src/ragbits/core/options.py                                                                17       0  100.00%
packages/ragbits-core/src/ragbits/core/types.py                                                                   9       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/__init__.py                                                         74       5  93.24%   42-45, 52-54
packages/ragbits-core/src/ragbits/core/audit/base.py                                                             44       1  97.73%   119
packages/ragbits-core/src/ragbits/core/audit/cli.py                                                              67       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                     4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                        20       2  90.00%   56, 69
packages/ragbits-core/src/ragbits/core/embeddings/exceptions.py                                                  17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/embeddings/litellm.py                                                     38      18  52.63%   78-112
packages/ragbits-core/src/ragbits/core/embeddings/noop.py                                                         8       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                           4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/base.py                                                              66       8  87.88%   61, 73, 174, 193-201, 205
packages/ragbits-core/src/ragbits/core/llms/exceptions.py                                                        17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/llms/factory.py                                                           12       3  75.00%   30, 41, 51
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                           95      29  69.47%   94, 100-121, 165, 175, 204-229, 251-256, 267
packages/ragbits-core/src/ragbits/core/llms/local.py                                                             56      27  51.79%   9-12, 66-74, 86-87, 108-119, 140-156
packages/ragbits-core/src/ragbits/core/llms/mock.py                                                              29       9  68.97%   63-74
packages/ragbits-core/src/ragbits/core/metadata_stores/__init__.py                                                3       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/base.py                                                   11       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/exceptions.py                                              4       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/in_memory.py                                              16       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                         2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/_cli.py                                                            44      21  52.27%   25-33, 47-49, 63-65, 73-75, 89-97
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                            20       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                         35       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                         126       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/__init__.py                                               2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/prompt_discovery.py                                      33       2  93.94%   54-55
packages/ragbits-core/src/ragbits/core/utils/__init__.py                                                          0       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/_pyproject.py                                                       38       1  97.37%   113
packages/ragbits-core/src/ragbits/core/utils/config_handling.py                                                  72       8  88.89%   16, 54-55, 62-63, 151-153
packages/ragbits-core/src/ragbits/core/utils/decorators.py                                                       29       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/dict_transformations.py                                             72       3  95.83%   24, 27, 108
packages/ragbits-core/src/ragbits/core/vector_stores/__init__.py                                                  3       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/_cli.py                                                     51       4  92.16%   63, 85, 91, 121
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                                                     40       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                                                   59       1  98.31%   77
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                                                38       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py                                                   62       1  98.39%   77
packages/ragbits-core/tests/cli/__init__.py                                                                       0       0  100.00%
packages/ragbits-core/tests/cli/test_cli_trace_handler.py                                                        48       3  93.75%   30, 43, 56
packages/ragbits-core/tests/cli/test_vector_store.py                                                            103       0  100.00%
packages/ragbits-core/tests/integration/vector_stores/test_vector_store.py                                       31       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                      0       0  100.00%
packages/ragbits-core/tests/unit/test_options.py                                                                 21       0  100.00%
packages/ragbits-core/tests/unit/audit/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/audit/test_cli.py                                                              102       0  100.00%
packages/ragbits-core/tests/unit/audit/test_trace.py                                                             93       3  96.77%   14, 17, 20
packages/ragbits-core/tests/unit/embeddings/test_from_config.py                                                  14       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                                 0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_from_config.py                                                        16       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                            70       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/__init__.py                                                         0       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_default_llm.py                                            12       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/__init__.py                                                      0       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_from_config.py                                             11       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_in_memory.py                                               22       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                              0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                         65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                         165       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                    0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                   30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                      18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                     2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py             3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py        14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py        14       0  100.00%
packages/ragbits-core/tests/unit/utils/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/utils/test_config_handling.py                                                   65       2  96.92%   27-28
packages/ragbits-core/tests/unit/utils/test_decorators.py                                                        26       2  92.31%   17, 39
packages/ragbits-core/tests/unit/utils/test_dict_transformations.py                                              69       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_find.py                                                    13       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_config.py                                               9       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_instace.py                                             37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/__init__.py                                                        0       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                                                    61       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_from_config.py                                               38       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                                                 84       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py                                                    40       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                          2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                           115       4  96.52%   257-264, 267
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                               61       2  96.72%   99, 146
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                                78      12  84.62%   87, 167-174, 183-185
packages/ragbits-document-search/src/ragbits/document_search/documents/exceptions.py                             11       5  54.55%   7-8, 17, 26-27
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py                               122      16  86.89%   131, 214-219, 256-259, 263-264, 289-291
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py                     33       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/__init__.py           5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/base.py              25       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/batched.py           18       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/distributed.py       30       8  73.33%   8-9, 36, 64-71
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/sequential.py        13       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                      3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                         19       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                        20       7  65.00%   33, 54-60
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/__init__.py         4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/default.py         46       4  91.30%   98, 103-104, 137
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/images.py          50      19  62.00%   73-80, 87-99, 111, 124
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/pdf.py             19       6  68.42%   23, 35-43
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/utils.py           38      11  71.05%   71, 82-83, 98-101, 110, 121-123
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                     5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                         9       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/llm.py                         25       4  84.00%   47-50
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                         6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/prompts.py                     16       1  93.75%   52
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                      3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                         17       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py                      16       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                          9       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                                0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                                 3       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                    0       0  100.00%
packages/ragbits-document-search/tests/integration/test_rerankers.py                                             15       6  60.00%   18-38
packages/ragbits-document-search/tests/integration/test_sources.py                                               23      10  56.52%   22-32, 40-45
packages/ragbits-document-search/tests/integration/test_unstructured.py                                          48      10  79.17%   52-58, 71-77
packages/ragbits-document-search/tests/unit/test_config.py                                                       57       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_processor.py                                           17       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                              88       0  100.00%
packages/ragbits-document-search/tests/unit/test_documents.py                                                    13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                                     19       0  100.00%
packages/ragbits-document-search/tests/unit/test_local_file_source.py                                            13       0  100.00%
packages/ragbits-document-search/tests/unit/test_processing_strategies.py                                        25       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                                    41       0  100.00%
packages/ragbits-document-search/tests/unit/test_rephrasers.py                                                   26       0  100.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                                    51       1  98.04%   23
packages/ragbits-document-search/tests/unit/test_source_discriminator.py                                         35       0  100.00%
packages/ragbits-document-search/tests/unit/test_sources.py                                                      25       0  100.00%
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/__init__.py                0       0  100.00%
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/factories.py              11       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/__init__.py                                                    0       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/base.py                                                       15       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/openai_moderation.py                                          19       5  73.68%   29-33
packages/ragbits-guardrails/tests/unit/test_openai_moderation.py                                                 35       0  100.00%
TOTAL                                                                                                          4274     320  92.51%

Diff against main

Filename                                                  Stmts    Miss  Cover
------------------------------------------------------  -------  ------  --------
packages/ragbits-core/src/ragbits/core/llms/base.py         +13      +1  +1.09%
packages/ragbits-core/src/ragbits/core/llms/litellm.py       +5      +1  +0.58%
packages/ragbits-core/tests/unit/llms/test_litellm.py        +7       0  +100.00%
TOTAL                                                       +25      +2  -0.01%

Results for commit: 1113b2b

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

github-actions · 2025-01-15T11:03:09Z

Trivy scanning results.

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: Slack (slack-web-hook)
════════════════════════════════════════
Slack Webhook
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/proxy/_types.py:1314
────────────────────────────────────────
1312 alert_to_webhook_url: Optional[Dict] = Field(
1313 None,
1314 [ bhook_url: {'budget_alerts': '*****************************************************************************'}`",
1315 )
────────────────────────────────────────

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: JWT (jwt-token)
════════════════════════════════════════
JWT token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA:80
────────────────────────────────────────
78 >>> encoded = jwt.encode({"some": "payload"}, "secret", algorithm="HS256")
79 >>> print(encoded)
80 [ *********************************************************************************************************
81 >>> jwt.decode(encoded, "secret", algorithms=["HS256"])
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/llms/huggingface/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

Total: 1 (MEDIUM: 0, HIGH: 0, CRITICAL: 1)

CRITICAL: HuggingFace (hugging-face-access-token)
════════════════════════════════════════
Hugging Face Access Token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/llms/huggingface/huggingface_llms_metadata/hf_text_generation_models.txt:36162
────────────────────────────────────────
36160 mncai/Llama2-7B-Active_3rd-floor-LoRA-dim64_epoch4
36161 ajcdp/CM
36162 [ Nagharjun17/*************************************
36163 BigSalmon/InformalToFormalLincoln114Paraphrase
────────────────────────────────────────

micpst reviewed Jan 15, 2025

View reviewed changes

packages/ragbits-core/src/ragbits/core/llms/base.py Show resolved Hide resolved

kdziedzic68 added 3 commits January 15, 2025 11:49

fix linters

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Learn about vigilant mode

Loading
Loading status checks…

cbfa6bd

fix tests and formatter

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Learn about vigilant mode

Loading
Loading status checks…

f78c0cd

rebase

773bf6b

kdziedzic68 force-pushed the feat/metadata-for-llm branch from f9a71e4 to 773bf6b Compare January 15, 2025 10:58

kdziedzic68 and others added 3 commits January 15, 2025 12:59

fixing mypy

731d9e4

fix typing

ef0503d

Merge branch 'main' into feat/metadata-for-llm

8608333

micpst linked an issue Jan 17, 2025 that may be closed by this pull request

feat: support metadata output from LLM #266

Closed

micpst added 3 commits January 17, 2025 19:01

fix validation error + add test

14681b9

linters

45cdcfd

types

1113b2b

micpst approved these changes Jan 17, 2025

View reviewed changes

micpst merged commit 4055f8d into main Jan 17, 2025
6 checks passed

micpst deleted the feat/metadata-for-llm branch January 17, 2025 18:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support metadata output from LLM #274

feat: support metadata output from LLM #274

kdziedzic68 commented Jan 15, 2025

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

feat: support metadata output from LLM #274

feat: support metadata output from LLM #274

Conversation

kdziedzic68 commented Jan 15, 2025

github-actions bot commented Jan 15, 2025 • edited Loading

Code Coverage Summary

Diff against main

github-actions bot commented Jan 15, 2025 • edited Loading

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

.venv/lib/python3.10/site-packages/litellm/llms/huggingface/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading