Spark notebooks quality check using Ollama LLM

1. • Install Ollama using - brew install ollama
2. • I used llama2 version . Pull llama2 version using below command - ollama pull llama2

3. • Run using - ollama run llama2

     

4. • I have spark running in docker. I opened Jupiter Notebook and added below code

Please note down few important point

# Call local Ollama API
ollama_url = "http://host.docker.internal:11434/api/generate"  - This code is consuming ollama model running in my local mac.

Steps for running jupiter notebook in Docker is given below

1. Install Docker for Windows/ Mac in your laptop ( I am using Mac)

2. I have provided docker-compose file in repository

3. In prompt execute command - docker-compose up -d

4. You will see

5. Open http://localhost:8888/ , you will get Jupyter notebook in browser

6. It will ask you enter a token to create password

7. You can get token from the below section - refer query string-

8. Once you logged in , you can upload notebook in the interface and test above scenarios