1. What is FastAPI?

FastAPI is a Python web framework used to build APIs. It is particularly useful for ML/LLM applications because it provides:

High performance
Automatic API documentation
Request/response validation
Easy integration with Python ML libraries
Asynchronous programming support

A typical LLM inference architecture looks like:

Client
   |
   | HTTP POST /generate
   v
FastAPI Server
   |
   v
Hugging Face Transformer Model
   |
   v
Generated Text
   |
   v
Client

2. Installing FastAPI

Create a virtual environment and install the required packages:

pip install fastapi uvicorn

For Hugging Face models:

pip install transformers torch

3. Creating a Basic FastAPI App

Create main.py:

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def home():
    return {
        "message": "LLM Inference API is running"
    }

Run the server:

uvicorn main:app --reload

The API will be available at:

http://127.0.0.1:8000

FastAPI also automatically provides interactive documentation at:

http://127.0.0.1:8000/docs

and:

http://127.0.0.1:8000/redoc

4. Creating API Endpoints

For example, create a simple greeting endpoint:

from fastapi import FastAPI

app = FastAPI()


@app.get("/hello")
def hello():
    return {
        "message": "Hello from FastAPI"
    }

Request:

GET /hello

Response:

{
    "message": "Hello from FastAPI"
}

5. POST Request with Input Data

LLM APIs normally receive prompts using POST. FastAPI uses Pydantic models to validate request data.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class PromptRequest(BaseModel):
    prompt: str


@app.post("/generate")
def generate(request: PromptRequest):
    return {
        "prompt": request.prompt,
        "response": "This is a generated response."
    }

Example request:

{
    "prompt": "Explain machine learning"
}

Response:

{
    "prompt": "Explain machine learning",
    "response": "This is a generated response."
}

6. Loading a Hugging Face Model

Hugging Face’s transformers library provides pretrained models and tokenizers. For demonstration, we can use a small text-generation model.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "distilgpt2"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

Here:

AutoTokenizer converts text into tokens.
AutoModelForCausalLM loads a causal language model.
distilgpt2 is a relatively small model suitable for demonstration.

For production, you would generally choose a model based on your hardware, latency, context length, licensing, and quality requirements.

7. Generating Text with the Hugging Face Model

The basic inference process is:

prompt = "Artificial intelligence is"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50
)

response = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True
)

print(response)

The process is:

Prompt
  ↓
Tokenizer
  ↓
Input Tokens
  ↓
Hugging Face Model
  ↓
Output Tokens
  ↓
Tokenizer
  ↓
Generated Text

8. Building the LLM Inference API

Now combine FastAPI and Hugging Face. main.py:

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM

app = FastAPI(
    title="LLM Inference API",
    description="FastAPI application serving a Hugging Face model",
    version="1.0.0"
)

MODEL_NAME = "distilgpt2"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)


class GenerateRequest(BaseModel):
    prompt: str
    max_new_tokens: int = 50


@app.get("/")
def home():
    return {
        "message": "LLM API is running"
    }


@app.post("/generate")
def generate(request: GenerateRequest):

    inputs = tokenizer(
        request.prompt,
        return_tensors="pt"
    )

    outputs = model.generate(
        **inputs,
        max_new_tokens=request.max_new_tokens
    )

    generated_text = tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )

    return {
        "prompt": request.prompt,
        "response": generated_text
    }

Run it:

uvicorn main:app --reload

9. Testing the API

Using curl:

curl -X POST "http://127.0.0.1:8000/generate" \
     -H "Content-Type: application/json" \
     -d '{
           "prompt": "Artificial intelligence is",
           "max_new_tokens": 50
         }'

Example response:

{
    "prompt": "Artificial intelligence is",
    "response": "Artificial intelligence is ..."
}

You can also test it interactively through:

http://127.0.0.1:8000/docs

10. Improving the Generation Parameters

Hugging Face’s generate() supports several useful parameters.

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

Important parameters:

Parameter	Purpose
`max_new_tokens`	Maximum number of newly generated tokens
`temperature`	Controls randomness
`top_p`	Nucleus sampling
`do_sample`	Enables sampling
`num_return_sequences`	Number of generated responses

For example:

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

11. Running the Model on GPU

For larger models, GPU inference is usually necessary.

import torch

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = model.to(device)

Then move the inputs to the same device:

inputs = tokenizer(
    request.prompt,
    return_tensors="pt"
)

inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}

Then:

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=request.max_new_tokens
    )

12. A Better Production-Oriented Version

You don’t want to repeatedly load the model for every request. Instead, load it once when the application starts.

from contextlib import asynccontextmanager

from fastapi import FastAPI
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM
)

MODEL_NAME = "distilgpt2"

tokenizer = None
model = None


@asynccontextmanager
async def lifespan(app: FastAPI):
    global tokenizer, model

    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)

    yield

    del model
    del tokenizer


app = FastAPI(lifespan=lifespan)

Then your endpoint can use the already-loaded model:

from pydantic import BaseModel


class GenerateRequest(BaseModel):
    prompt: str
    max_new_tokens: int = 50


@app.post("/generate")
def generate(request: GenerateRequest):

    inputs = tokenizer(
        request.prompt,
        return_tensors="pt"
    )

    outputs = model.generate(
        **inputs,
        max_new_tokens=request.max_new_tokens
    )

    response = tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )

    return {
        "response": response
    }

This avoids loading the model on every API request.

13. Using a Chat-Style API

For an LLM application, you may want an API resembling a chat endpoint. Request:

{
    "messages": [
        {
            "role": "user",
            "content": "What is machine learning?"
        }
    ]
}

Pydantic models can represent this:

from typing import List
from pydantic import BaseModel


class Message(BaseModel):
    role: str
    content: str


class ChatRequest(BaseModel):
    messages: List[Message]

Endpoint:

@app.post("/chat")
def chat(request: ChatRequest):

    prompt = request.messages[-1].content

    inputs = tokenizer(
        prompt,
        return_tensors="pt"
    )

    outputs = model.generate(
        **inputs,
        max_new_tokens=100
    )

    response = tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )

    return {
        "message": {
            "role": "assistant",
            "content": response
        }
    }

14. Project Structure

A small LLM API can be organized like this:

llm-api/
│
├── app/
│   ├── __init__.py
│   ├── main.py
│   ├── model.py
│   └── schemas.py
│
├── requirements.txt
└── README.md

`schemas.py`

from pydantic import BaseModel


class GenerateRequest(BaseModel):
    prompt: str
    max_new_tokens: int = 50

`model.py`

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM
)

MODEL_NAME = "distilgpt2"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)


def generate_text(prompt: str, max_new_tokens: int = 50):

    inputs = tokenizer(
        prompt,
        return_tensors="pt"
    )

    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens
    )

    return tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )

`main.py`

from fastapi import FastAPI

from .schemas import GenerateRequest
from .model import generate_text

app = FastAPI()


@app.get("/")
def home():
    return {"message": "LLM API running"}


@app.post("/generate")
def generate(request: GenerateRequest):

    response = generate_text(
        request.prompt,
        request.max_new_tokens
    )

    return {
        "response": response
    }

Run:

uvicorn app.main:app --reload

15. Production Architecture

For a real LLM deployment, the architecture is usually closer to:

                    ┌──────────────┐
                    │    Client    │
                    └──────┬───────┘
                           │
                         HTTP
                           │
                    ┌──────▼───────┐
                    │   FastAPI    │
                    │     API      │
                    └──────┬───────┘
                           │
                    ┌──────▼───────┐
                    │ Model Server │
                    │              │
                    │ Hugging Face │
                    │ Transformer  │
                    └──────┬───────┘
                           │
                       GPU / CPU

For larger production LLMs, you may use a dedicated inference engine rather than directly calling model.generate() inside a FastAPI process. Examples include Hugging Face’s Text Generation Inference (TGI) and other optimized serving frameworks.

16. Key Concepts to Remember

FastAPI

FastAPI
  ├── Routes
  ├── Request validation
  ├── Response serialization
  ├── Async support
  └── Automatic API documentation

Hugging Face inference

Text
 ↓
Tokenizer
 ↓
Tensor
 ↓
Model
 ↓
Generated Tokens
 ↓
Tokenizer
 ↓
Text

LLM API

POST /generate
        ↓
   JSON Request
        ↓
     FastAPI
        ↓
  Hugging Face Model
        ↓
   JSON Response

Most important practical point: load the model once, not once per request; keep model/tokenizer initialization outside the request handler, use the appropriate device (cuda when available), and for large production models consider a specialized inference server rather than running raw Transformers generation directly inside FastAPI.

​1. What is FastAPI?

​2. Installing FastAPI

​3. Creating a Basic FastAPI App

​4. Creating API Endpoints

​5. POST Request with Input Data

​6. Loading a Hugging Face Model

​7. Generating Text with the Hugging Face Model

​8. Building the LLM Inference API

​9. Testing the API

​10. Improving the Generation Parameters

​11. Running the Model on GPU

​12. A Better Production-Oriented Version

​13. Using a Chat-Style API

​14. Project Structure

​schemas.py

​model.py

​main.py

​15. Production Architecture

​16. Key Concepts to Remember

​FastAPI

​Hugging Face inference

​LLM API