% python ask.py -i local -c -q "How does Ask.py work?"
2024-11-20 10:00:09,335 - INFO - Initializing converter ...
2024-11-20 10:00:09,335 - INFO - ✅ Successfully initialized Docling.
2024-11-20 10:00:09,335 - INFO - Initializing chunker ...
2024-11-20 10:00:09,550 - INFO - ✅ Successfully initialized Chonkie.
2024-11-20 10:00:09,850 - INFO - Initializing database ...
2024-11-20 10:00:09,933 - INFO - ✅ Successfully initialized DuckDB.
2024-11-20 10:00:09,933 - INFO - Processing the local data directory ...
2024-11-20 10:00:09,933 - INFO - Processing README.pdf ...
Fetching 9 files: 100%|████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 11781.75it/s]
2024-11-20 10:00:29,629 - INFO - ✅ Finished processing README.pdf.
2024-11-20 10:00:29,629 - INFO - Chunking the text ...
2024-11-20 10:00:29,639 - INFO - ✅ Generated 2 chunks ...
2024-11-20 10:00:29,639 - INFO - Saving 2 chunks to DB ...
2024-11-20 10:00:29,681 - INFO - Embedding 1 batches of chunks ...
2024-11-20 10:00:30,337 - INFO - ✅ Finished embedding.
2024-11-20 10:00:30,423 - INFO - ✅ Created the vector index ...
2024-11-20 10:00:30,483 - INFO - ✅ Created the full text search index ...
2024-11-20 10:00:30,483 - INFO - ✅ Successfully embedded and saved chunks to DB.
2024-11-20 10:00:30,483 - INFO - Querying the vector DB to get context ...
2024-11-20 10:00:30,773 - INFO - Running full-text search ...
2024-11-20 10:00:30,796 - INFO - ✅ Got 2 matched chunks.
2024-11-20 10:00:30,797 - INFO - Running inference with context ...
2024-11-20 10:00:34,939 - INFO - ✅ Finished inference API call.
2024-11-20 10:00:34,939 - INFO - Generating output ...
# Answer
Ask.py is a Python program designed to implement a search-extract-summarize flow, similar to AI search engines like Perplexity. It can be run through a command line interface or a GradIO user interface and allows for flexibility in controlling output and search behaviors[1].
When a query is executed, Ask.py performs the following steps:
1. Searches Google for the top 10 web pages related to the query.
2. Crawls and scrapes the content of these pages.
3. Breaks down the scraped text into chunks and saves them in a vector database.
4. Conducts a vector search with the initial query to identify the top 10 matched text chunks.
5. Optionally integrates full-text search results and uses a reranker to refine the results.
6. Utilizes the selected chunks as context to query a language model (LLM) to generate a comprehensive answer.
7. Outputs the answer along with references to the sources[1].
Moreover, the program allows various configurations such as date restrictions, site targeting, output language, and output length. It can also scrape specified URL lists instead of performing a web search, making it highly versatile for search and data extraction tasks[2].
# References
[1] file:///Users/feng/work/github/ask.py/data/README.pdf
[2] file:///Users/feng/work/github/ask.py/data/README.pdf