namuan/dr-doc-search

This is stars and forks stats for /namuan/dr-doc-search repository. As of 26 Apr, 2024 this repository has 593 stars and 61 forks.

Doc Search Converse with a book (PDF) See tweet for full demo. Documentation: https://namuan.github.io/dr-doc-search Source Code: https://github.com/namuan/dr-doc-search PyPI: https://pypi.org/project/dr-doc-search/ Pre-requisites Tessaract OCR ImageMagick Note: If you are using Windows, then make sure that you set the location of ImageMagick executable in the IMCONV environment variable. # For example, if you have installed ImageMagick in PROGRAMFILES\ImageMagick-7.1.0-Q16-HDRI set IMCONV="%PROGRAMFILES%\ImageMagick-7.1.0-Q16-HDRI\magick" Installation pip install dr-doc-search Example Usage There are two steps to use this application: 1. First, you need to create the index and generate embeddings for the PDF file. Here I'm using a PDF file generated from this page Parable of a Monetary Economy Before running this, you need to set up your OpenAI API key. You can get it from OpenAI. From version 1.5.0, you can skip OpenAI and use HuggingFace models to generate embeddings and answers. export OPENAI_API_KEY=<your-openai-api-key> The run the following command to start the training process: dr-doc-search --train -i ~/Downloads/parable-of-a-monetary-economy-heteconomist.pdf Use huggingface for generating embeddings: dr-doc-search --train -i ~/Downloads/parable-of-a-monetary-economy-heteconomist.pdf --embedding huggingface The training process generates some temporary files in the OutputDir/dr-doc-search/<pdf-name> folder under your home directory. Here is what it looks like: ~/OutputDir/dr-doc-search/parable-of-a-monetary-economy-heteconomist $ tree . ├── images │ ├── output-1.png │ ├── output-10.png │ ├── output-11.png ... │ └── output-9.png ├── index │ ├── docsearch.index │ └── index.pkl ├── parable-of-a-monetary-economy-heteconomist.pdf └── scanned ├── output-1.txt ... └── output-9.txt Note: It is possible to change the base of the output directory by providing the --app-dir argument. 2. Now that we have the index, we can use it to start asking questions. dr-doc-search -i ~/Downloads/parable-of-a-monetary-economy-heteconomist.pdf --input-question "How did the attempt to reduce the debut resulted in decrease in employment?" Or You can open up a web interface (on port :5006) to ask questions: dr-doc-search --web-app -i ~/Downloads/parable-of-a-monetary-economy-heteconomist.pdf To use huggingface model, provide the --llm argument: dr-doc-search --web-app -i ~/Downloads/parable-of-a-monetary-economy-heteconomist.pdf --llm huggingface There are more options for choose the start and end pages for the PDF file. See the help for more details: dr-doc-search --help Acknowledgements anton/@abacaj for the idea LangChain HoloViz Panel OpenAI Development Clone this repository Requirements: Python 3.7+ Poetry Create a virtual environment and install the dependencies poetry install Activate the virtual environment poetry shell Validating build make build Release process A release is automatically published when a new version is bumped using make bump. See .github/workflows/build.yml for more details. Once the release is published, .github/workflows/publish.yml will automatically publish it to PyPI. Disclaimer This project is not affiliated with OpenAI. The OpenAI API and GPT-3 language model are not free after the trial period.