Generative AI : How to run a LLM with Python — Testcontainers and Ollama

Bring your ideas to live with Gen AI and Python. A straightforward guide to run the LLM of your choice.

Testing a Generative AI or Large Language Models (LLM) is crucial to ensure they behave as expected under various conditions. This tutorial will guide you through the process of testing LLM and prompts using Testcontainers and Ollama in Python, enabling robust and repeatable testing environments.

Testcontainers, that runs Docker, that runs Ollama that runs LLMs(image made by author)

Quick definitions

LLM is a large language model, a type of neural network that can generate human-like text.

Testcontainers is an open source framework for providing any service that can run in a Docker container to run integration tests in your favorite language.

Ollama is a user-friendly, fast, and lightweight tool designed to run large language models (LLM) locally on your computer.

Prerequisites

To follow this tutorial, you’ll need:

• Basic knowledge of Python programming
• Docker installed on your machine
• Python(3.8+) environment set up with the required libraries

Let’s get started!

Step-by-Step Guide

1. Set up your environment

First, ensure you have the necessary Python packages installed. You can use pip to install them:

• Create a python virtual environment (optional but recommended)

virtualenv .venv

• Activate the virtual environment (optional but recommended)

source .venv/bin/activate

• Install the required packages

pip install requests testcontainers pytest

2. Define your system template

The system template is crucial as it sets the context for the LLM. Here’s an example template used to redirect questions to the appropriate contact within a company:

system_template = """
Here's an improved version of the prompt:
You…