# Account Management
# Account Management
Your credits, tokens, and keys. The administrative sacraments.
## Commands
| Command | Description |
| ----------------- | ------------------------ |
| `balance` | Check credit balance |
| `fund` | Show TAO deposit address |
| `fund list` | List deposit history |
| `tokens create` | Create API token |
| `tokens list` | List API tokens |
| `tokens revoke` | Revoke API token |
| `ssh-keys add` | Register SSH public key |
| `ssh-keys list` | List registered SSH keys |
| `ssh-keys delete` | Delete SSH key |
## API Tokens
API tokens provide programmatic access for CI/CD pipelines, automation, and server deployments.
Save your token immediately after creation. The full token is only displayed once and cannot be retrieved later.
Set the token via environment variable: `export BASILICA_API_TOKEN=basilica_xxxxx`
## SSH Keys
SSH keys are used to access GPU rental instances. The CLI auto-detects keys from `~/.ssh/` in this order:
1. `id_ed25519.pub`
2. `id_ecdsa.pub`
3. `id_rsa.pub`
## Credits & Funding
See [Core Concepts](/introduction/concepts#credits) for details on how credits work.
Quick start: Run `fund` to get your TAO deposit address, then send TAO from your Bittensor wallet. Credits are added after 12 block confirmations. Track status with `fund list`.
## Security Best Practices
### API Tokens
* Use descriptive names (e.g., `github-ci`, `production-server`)
* One token per service
* Rotate regularly
* Use environment variables, never hardcode
* Revoke unused tokens
## Troubleshooting
### Insufficient balance
Check balance and fund your account. Use `fund list` to check pending deposits.
### Invalid token
Verify token is set correctly with `echo $BASILICA_API_TOKEN`. Check if token exists with `tokens list`. Create a new token if needed.
### SSH key not working
Verify key is registered with `ssh-keys list`. Check local key exists in `~/.ssh/`. Re-add if needed.
## Next Steps
* **[GPU Rentals](/cli/gpu-rentals):** Rent and manage GPU instances
* **[Deployments](/cli/deployments):** Deploy applications
* **[Core Concepts](/introduction/concepts):** Understand pricing and billing
# CLI Authentication
# CLI Authentication
Two ways in: browser OAuth or device code flow for headless environments.
For API tokens and SDK auth, see [Authentication](/introduction/authentication).
## Login Flows
### Standard Flow
The `login` command opens your default browser to the Basilica auth page. After signing in and authorizing the CLI, credentials are stored locally.
### Device Flow
For headless environments (WSL, SSH sessions, containers), use `--device-code`. The CLI displays a URL and code to enter in any browser.
## Token Storage
Credentials are stored in `~/.local/share/basilica/auth.json` containing:
* Access token (for API requests)
* Refresh token (for automatic renewal)
* Token expiration time
Never share or commit your auth.json file.
## Session Management
Tokens are automatically refreshed when they expire. No manual action required.
To switch accounts, run `logout` then `login` again.
## Troubleshooting
### Browser doesn't open
Use device flow: `basilica login --device-code`
### Failed to store tokens
Check permissions: `ls -la ~/.local/share/basilica/`
Ensure the directory exists and is writable:
```bash
mkdir -p ~/.local/share/basilica
chmod 700 ~/.local/share/basilica
```
### Device flow timeout
The code expires after 15 minutes. Run `basilica login --device-code` again for a new code.
## Next Steps
* **[GPU Rentals](/cli/gpu-rentals):** Rent GPU instances
* **[Deployments](/cli/deployments):** Deploy applications
* **[Account Management](/cli/account):** Manage tokens and SSH keys
# CLI Deployments
# CLI Deployments
Ship code from the command line. Python files, Docker images, LLM servers.
## Commands
| Command | Description |
| --------------- | ------------------------------------ |
| `deploy` | Deploy a Python file or Docker image |
| `deploy ls` | List all deployments |
| `deploy status` | Get detailed deployment status |
| `deploy logs` | Stream deployment logs |
| `deploy scale` | Scale deployment replicas |
| `deploy delete` | Delete a deployment |
| `deploy vllm` | Deploy vLLM inference server |
| `deploy sglang` | Deploy SGLang inference server |
## Deployment Sources
The `deploy` command accepts:
* **Python files** (`.py`): Deployed with a configurable base image
* **Docker images:** Deployed directly
## Health Checks
Configure liveness, readiness, and startup probes to ensure your application runs correctly. The CLI supports HTTP path probes with configurable timing.
Key concepts:
* **Liveness probe**: Restarts container if unhealthy
* **Readiness probe**: Controls traffic routing
* **Startup probe**: For slow-starting applications
Use `--health-path` as shorthand for all probes, or configure each individually.
## LLM Inference Servers
### vLLM
Deploy OpenAI-compatible inference servers with vLLM. Supports tensor parallelism, quantization (AWQ, GPTQ, FP8), and custom model configurations.
### SGLang
Alternative inference server with similar capabilities and SGLang-specific optimizations.
Both commands auto-detect GPU requirements based on model size when not specified.
## Troubleshooting
### Deployment stuck in Pending
1. Check status with `--show-phases` for details
2. Check logs for errors
3. Verify GPU availability with `ls --compute citadel`
### Container crash loop
Common causes:
* Application error on startup
* Missing environment variables
* Port binding issues
Check logs for specific errors.
### Health check failures
1. Ensure health endpoint responds with 200 OK
2. Increase initial delay for slow-starting apps
3. Verify port matches your application
## Next Steps
* **[LLM Inference](/inference):** vLLM and SGLang deployment guides
* **[Account Management](/cli/account):** Billing and API tokens
* **[Python SDK](/sdk/deployments):** Programmatic deployments
# GPU Rentals
# GPU Rentals
Raw GPU access via SSH. Browse what's available, spin one up, SSH in, do your thing.
## Commands
| Command | Description |
| --------- | ---------------------------- |
| `ls` | List available GPU instances |
| `up` | Start a GPU rental |
| `ps` | List active rentals |
| `status` | Check detailed rental status |
| `logs` | View instance logs |
| `down` | Terminate a rental |
| `restart` | Restart instance container |
| `exec` | Execute commands on instance |
| `ssh` | SSH into instance |
| `cp` | Copy files to/from instance |
## Interactive Selection
When multiple GPU options match your criteria, the CLI presents an interactive selector to choose from available offerings.
## Infrastructure Sources
Filter by infrastructure source using `--compute citadel` or `--compute bourse`. See [Core Concepts](/introduction/concepts) for details.
## File Transfer Path Format
The `cp` command uses `RENTAL_ID:/path` format for remote paths:
* Local to remote: `basilica cp ./file.txt RENTAL_ID:/workspace/file.txt`
* Remote to local: `basilica cp RENTAL_ID:/workspace/results.csv ./results.csv`
## Troubleshooting
### No GPUs available
Try different GPU types, regions, or cloud types.
### SSH connection failed
1. Check rental status with `status` command
2. Verify SSH key is registered with `ssh-keys list`
3. Wait 1-2 minutes for instance to be ready
### File transfer slow
For large files, consider compressing before transfer or using cloud storage (S3, GCS) for large datasets.
## Next Steps
* **[Deployments](/cli/deployments):** Deploy applications
* **[Account Management](/cli/account):** Manage billing and keys
* **[LLM Inference](/inference):** Deploy LLM servers
# Overview
# Overview
For those who prefer the terminal. Spin up GPUs, deploy code, manage your account. No GUI required, no friction added.
## Installation
### macOS / Linux
```bash
curl -sSL https://basilica.ai/install.sh | bash
```
Verify with `basilica --version`.
## Commands
### Authentication
| Command | Description |
| -------------------------------------- | -------------------------- |
| [`login`](/cli/authentication#login) | Authenticate with Basilica |
| [`logout`](/cli/authentication#logout) | Clear stored credentials |
### GPU Rentals
| Command | Description |
| ------------------------------------- | ---------------------------- |
| [`ls`](/cli/gpu-rentals#ls) | List available GPU instances |
| [`up`](/cli/gpu-rentals#up) | Start a GPU rental |
| [`ps`](/cli/gpu-rentals#ps) | List active rentals |
| [`status`](/cli/gpu-rentals#status) | Check rental status |
| [`logs`](/cli/gpu-rentals#logs) | View instance logs |
| [`down`](/cli/gpu-rentals#down) | Terminate a rental |
| [`restart`](/cli/gpu-rentals#restart) | Restart instance container |
| [`exec`](/cli/gpu-rentals#exec) | Execute commands on instance |
| [`ssh`](/cli/gpu-rentals#ssh) | SSH into instance |
| [`cp`](/cli/gpu-rentals#cp) | Copy files to/from instance |
### Deployments
| Command | Description |
| ------------------------------------------------- | ---------------------- |
| [`deploy`](/cli/deployments#deploy) | Deploy an application |
| [`deploy ls`](/cli/deployments#deploy-ls) | List deployments |
| [`deploy status`](/cli/deployments#deploy-status) | Get deployment status |
| [`deploy logs`](/cli/deployments#deploy-logs) | Stream deployment logs |
| [`deploy scale`](/cli/deployments#deploy-scale) | Scale replicas |
| [`deploy delete`](/cli/deployments#deploy-delete) | Delete a deployment |
| [`deploy vllm`](/cli/deployments#deploy-vllm) | Deploy vLLM server |
| [`deploy sglang`](/cli/deployments#deploy-sglang) | Deploy SGLang server |
### Account Management
| Command | Description |
| ------------------------------------- | -------------------- |
| [`balance`](/cli/account#balance) | Check credit balance |
| [`fund`](/cli/account#fund) | Show deposit address |
| [`fund list`](/cli/account#fund-list) | List deposit history |
| [`tokens`](/cli/account#tokens) | Manage API tokens |
| [`ssh-keys`](/cli/account#ssh-keys) | Manage SSH keys |
Run `basilica --help` for detailed options.
## Configuration
Config lives in `~/.config/basilica/`. Auth data in `~/.local/share/basilica/`.
| Environment Variable | Description |
| -------------------- | ---------------------------- |
| `BASILICA_API_TOKEN` | API token for authentication |
| `BASILICA_API_URL` | API endpoint URL |
## Infrastructure Sources
| Name | Description | Best For |
| --------------- | ------------------------------------------------------------ | --------------------- |
| **The Bourse** | Coming soon: community cloud for decentralized compute nodes | Future compute supply |
| **The Citadel** | Datacenter infrastructure with SLAs | Production workloads |
Filter by source using `--compute bourse` or `--compute citadel`.
## Auto-Login
Not logged in? No problem. Commands that need auth will prompt you automatically.
## Exit Codes
| Code | Meaning |
| ---- | -------------------- |
| `0` | Success |
| `1` | General error |
| `2` | Authentication error |
| `3` | Network error |
| `4` | Resource not found |
## Next Steps
* **[Authentication](/cli/authentication):** Login and credential management
* **[GPU Rentals](/cli/gpu-rentals):** Rent and manage GPU instances
* **[Deployments](/cli/deployments):** Deploy applications
* **[Account Management](/cli/account):** Billing and account settings
# Authentication
# Authentication
## CLI Login
Enter the Basilica:
```bash
basilica login
```
Opens a browser. Sign in. You're in.
### If Browser Callback Fails
If the browser callback doesn't work (common in remote environments or WSL), use the device code flow:
```bash
basilica login --device-code
```
This displays a code and URL. Open the URL in any browser, enter the code, and you're authenticated.
## Programmatic Access
For SDK usage, CI/CD pipelines, or automation, create an API token.
### Generate a Token
First, ensure you're logged in:
```bash
basilica login
```
Then create an API token:
```bash
basilica tokens create
```
Copy the token immediately. It won't be shown again.
### Use the Token
**With the SDK:**
```python
from basilica import BasilicaClient
client = BasilicaClient(api_key="your_token")
```
**With an environment variable:**
```bash
export BASILICA_API_TOKEN=your_token
```
The SDK and CLI automatically use `BASILICA_API_TOKEN` when set:
```python
from basilica import BasilicaClient
# Automatically uses BASILICA_API_TOKEN
client = BasilicaClient()
```
***
### Authentication Priority
When multiple auth methods are configured, Basilica uses this order:
1. `api_key` parameter passed to `BasilicaClient()`
2. `BASILICA_API_TOKEN` environment variable
3. Stored credentials from `basilica login`
### Revoke a Token
```bash
basilica tokens revoke
```
Takes effect immediately. No grace period.
## Next Steps
* **[Getting Started](/introduction/getting-started):** Deploy your first application
* **[Python SDK](/sdk):** Learn the full SDK API
* **[CLI Reference](/cli):** Complete CLI documentation
# Core Concepts
# Core Concepts
Everything you need to know before you start. About two minutes.
## Rentals
Grab a GPU or CPU and make it yours. Full SSH access, dedicated machine. Run whatever you want, however you want. It's your box.
## Deployments
Ship your code, get a public URL. Tell us what resources you need. We handle hosting, scaling, HTTPS. You pay per minute for what you use.
## Infrastructure Sources
Your rentals and deployments run on one of two infrastructure pools:
* **The Bourse:** Coming soon: community cloud for decentralized compute nodes.
* **The Citadel:** Datacenter partners with SLA guarantees. Fixed pricing, enterprise reliability.
## Credits
Everything on Basilica runs on credits. 1 credit = $1 USD. Simple as that.
### Fund Your Account
Deposit TAO to get credits:
1. Get your unique deposit address from the [CLI](/cli/account#credits--funding)
2. Send TAO from your wallet
3. Wait for confirmation (usually under 10 minutes)
4. Credits appear automatically at the current TAO/USD rate
If TAO is at $400 and you send 1 TAO, you'll have 400 credits ready to spend.
## Next Steps
* **[Getting Started](/introduction/getting-started):** Deploy something in the next few minutes
* **[Python SDK](/sdk):** Build with Basilica programmatically
* **[CLI Reference](/cli):** Manage everything from your terminal
# Quick Start
# Quick Start
Your first deployment, step by step. Five minutes from now, you'll have a live web application with a public URL.
## What We'll Build
A simple FastAPI web server that:
* Responds to HTTP requests
* Runs on Basilica infrastructure
* Has a public URL you can share
## Prerequisites
Before starting, ensure you have:
* Python 3.9+ installed
* Basilica CLI installed
* Basilica SDK installed (`pip install basilica-sdk`)
* Authenticated with Basilica (`basilica login`)
If you haven't completed these steps, see [Installation](/introduction/installation) first.
## Step 1: Create Your Project
Create a new directory for your project:
```bash
mkdir hello-basilica
cd hello-basilica
```
## Step 2: Write the Deployment Script
Create a file called `deploy.py`:
```python
import basilica
@basilica.deployment(
name="hello-world",
port=8000,
pip_packages=["fastapi", "uvicorn"],
)
def serve():
from fastapi import FastAPI
from datetime import datetime
import uvicorn
app = FastAPI(title="Hello Basilica")
@app.get("/")
def index():
return {
"message": "Hello from Basilica!",
"timestamp": datetime.now().isoformat()
}
@app.get("/health")
def health():
return {"status": "healthy"}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
print(f"URL: {deployment.url}")
```
## Step 3: Deploy
Run it:
```bash
python deploy.py
```
You'll see output like:
```
Deploying application...
✓ Deployed successfully!
Name: hello-world
URL: https://5c3d6478-f62f-4349-8954-cb89e8db1e7c.basilica.ai
State: running
```
## Step 4: Test Your Deployment
Open the URL in your browser, or use curl:
```bash
curl https://5c3d6478-f62f-4349-8954-cb89e8db1e7c.basilica.ai
```
Response:
```json
{
"message": "Hello from Basilica!",
"timestamp": "2024-01-15T10:30:00.123456"
}
```
Test the health endpoint:
```bash
curl https://5c3d6478-f62f-4349-8954-cb89e8db1e7c.basilica.ai/health
```
Response:
```json
{
"status": "healthy"
}
```
## Step 5: Monitor Your Deployment
### View Logs
Stream logs from your deployment:
```bash
basilica deploy logs 5c3d6478-f62f-4349-8954-cb89e8db1e7c
```
Or in Python:
```python
for log in client.logs("5c3d6478-f62f-4349-8954-cb89e8db1e7c"):
print(log)
```
### Check Status
Check deployment status:
```python
deployment = client.get("5c3d6478-f62f-4349-8954-cb89e8db1e7c")
print(f"State: {deployment.state}")
print(f"URL: {deployment.url}")
print(f"Created: {deployment.created_at}")
```
### List All Deployments
```python
for d in client.list():
print(f"{d.name}: {d.state}")
```
## Step 6: Update Your Deployment
To update the application, deploy again with the same name:
```python
updated_code = """
from fastapi import FastAPI
from datetime import datetime
app = FastAPI(title="Hello Basilica v2")
@app.get("/")
def index():
return {
"message": "Hello from Basilica v2!",
"version": "2.0.0",
"timestamp": datetime.now().isoformat()
}
@app.get("/health")
def health():
return {"status": "healthy", "version": "2.0.0"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
"""
deployment = client.deploy(
name="hello-world", # Same name updates the deployment
source=updated_code,
port=8000,
pip_packages=["fastapi", "uvicorn"],
)
print(f"Updated! URL: {deployment.url}")
```
## Step 7: Clean Up
Done experimenting? Delete the deployment to stop billing:
```python
client.delete("5c3d6478-f62f-4349-8954-cb89e8db1e7c")
print("Deployment deleted!")
```
Or via CLI:
```bash
basilica deploy down 5c3d6478-f62f-4349-8954-cb89e8db1e7c
```
## Alternative: Deploy from File
Instead of using the decorator, you can deploy existing files with `client.deploy()`:
**app.py:**
```python
from fastapi import FastAPI
import uvicorn
app = FastAPI()
@app.get("/")
def hello():
return {"message": "Hello from file!"}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
```
**deploy.py:**
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="hello-from-file",
source="app.py",
port=8000,
pip_packages=["fastapi", "uvicorn"],
)
print(f"URL: {deployment.url}")
```
Learn more in [Deployments](/sdk/deployments).
## Troubleshooting
### "Deployment failed" Error
Check the logs for details:
```bash
basilica deploy logs hello-world
```
Common causes:
* Syntax errors in your code
* Missing dependencies
* Port conflicts
### "Name already exists" Error
Each deployment name must be unique. Either:
* Use a different name
* Delete the existing deployment first
### Application Not Responding
1. Verify your app listens on `0.0.0.0`, not `127.0.0.1`
2. Confirm the port matches what you specified
3. Check logs for startup errors
### Slow Startup
First deployments may take longer as dependencies are installed. Subsequent deployments use cached dependencies.
## Next Steps
You've deployed your first application. Go deeper:
* **[Python SDK](/sdk):** Learn more SDK features
* **[Storage](/sdk/storage):** Add persistent storage
* **[GPU Configuration](/sdk/gpu):** Run on GPU hardware
* **[LLM Inference](/inference):** Deploy language models
# Enter the Basilica
# Enter the Basilica
Basilica is on-demand compute for AI. GPUs and CPUs, ready when you need them.
Basilica aggregates datacenter supply at market rates, with more infrastructure sources coming soon.
No gatekeepers. No waitlists. No genuflecting before cloud providers for quota increases.
Deploy containerized applications, serve LLMs, or claim raw compute nodes via SSH. Pay by the minute. Leave whenever you want.
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="my-model",
source="app.py",
gpu_count=1,
gpu_models=["H100"],
)
print(f"Blessed with a URL: {deployment.url}")
```
That's it. Your code is running on an H100.
## Two Ways to Compute
### Rentals
Grab a GPU node via SSH. Full root access, your container, your rules. Perfect for training runs, interactive development, or anything where you want direct control.
### Managed Deployments
Ship your code, get a public URL. We handle containers, HTTPS, and routing. Your app runs, you pay per minute.
Both access models draw from the same underlying infrastructure.
## Where the Compute Comes From
Every rental and deployment runs on one of two infrastructure sources:
### The Bourse
Coming soon. The Bourse is Basilica's community cloud for decentralized compute nodes.
### The Citadel
Curated datacenter partners: DataCrunch, Lambda, Hyperstack, HydraHost. Fixed pricing, SLA guarantees, enterprise reliability. The fortress for production workloads.
*Why the names? Basilica uses Knights Templar naming. The Bourse was the medieval merchant exchange. The Citadel, the fortified stronghold.*
## Why Developers Choose Basilica
**No YAML confessionals.** Your infrastructure is defined in Python, not sprawling configuration files.
**Pay for what you use.** Per-minute billing. Spin up an H100 for a few minutes of inference, pay for a few minutes.
**No quota purgatory.** Access A100s, H100s, and B200s without begging for limit increases or waiting for approval.
**More supply paths coming soon.** Basilica is expanding the ways compute reaches developers while keeping the same simple rental and deployment experience.
## The Essentials
| Concept | What It Means |
| -------------- | ---------------------------------------------------------------------- |
| **Deployment** | Your application running on Basilica. Gets a public URL automatically. |
| **Rental** | Direct compute access via SSH. For when you need raw control. |
| **Credits** | Your account balance. Funded by TAO deposits. |
## What Can You Build?
### LLM Inference
Run Llama, Mistral, or your fine-tuned models. vLLM or SGLang, your choice. OpenAI-compatible APIs out of the box.
### ML Model Serving
PyTorch, TensorFlow, Hugging Face. GPU-accelerated, publicly accessible.
### Web Applications
FastAPI, Flask, Gradio, Streamlit. If it runs in a container, it gets a public HTTPS URL. Simple as that.
### Batch Processing
Train models, process datasets, generate embeddings. Jobs clean up after themselves.
### Development Environments
Spin up in seconds. Tear down when you're done. No commitments, no lingering bills.
## Begin Your Journey
1. **[Installation](/introduction/installation):** Install the CLI and SDK
2. **[Authentication](/introduction/authentication):** Get your credentials
3. **[Getting Started](/introduction/getting-started):** Deploy your first application
4. **[Core Concepts](/introduction/concepts):** Understand the fundamentals
## Join the Community
The faithful gather here:
* **GitHub**: [one-covenant/basilica](https://github.com/one-covenant/basilica)
* **Discord**: [Join the congregation](https://discord.gg/eth2rJqYhr)
* **X**: [@basilic\_ai](https://x.com/basilic_ai)
# Installation
# Installation
Two ways to interact with Basilica: a Python SDK and a CLI. Install one or both.
**Python SDK:** For programmatic control from your Python applications. **CLI:** For terminal devotees and shell scripts.
## Python SDK
The recommended way to deploy and manage workloads. Write Python, get compute.
### Requirements
* Python 3.9 or higher
* pip or uv package manager
### Install with pip
```bash
pip install basilica-sdk
```
### Install with uv (Recommended)
```bash
uv pip install basilica-sdk
```
### Verify Installation
```python
import basilica
print(basilica.__version__)
```
## CLI
A standalone binary for managing rentals and deployments from the terminal. Fast and scriptable.
### macOS / Linux
```bash
curl -sSL https://basilica.ai/install.sh | bash
```
This script downloads the latest release and installs it to `/usr/local/bin/basilica`.
### From Source (Cargo)
For those who like to build from source. Useful for unreleased features or contributing to the project.
```bash
git clone https://github.com/one-covenant/basilica
cd basilica
cargo build --release -p basilica-cli
# Binary at ./target/release/basilica
```
### Verify CLI Installation
```bash
basilica --version
```
## Next Steps
You're in. Now authenticate:
* **[Authentication](/introduction/authentication):** Log in to your account
* **[Quick Start Guide](/introduction/getting-started):** Deploy your first application
# Custom Models
# Custom Models
Your fine-tuned creations deserve a home. Deploy from Hugging Face, local files, or private registries.
## Hugging Face Models
### Public Models
Deploy any public Hugging Face model:
```python
from basilica import BasilicaClient
client = BasilicaClient()
# vLLM deployment
deployment = client.deploy_vllm(
name="my-model",
model="your-username/your-model",
gpu_count=1,
)
# SGLang deployment
deployment = client.deploy_sglang(
name="my-model",
model="your-username/your-model",
gpu_count=1,
)
```
### Private Models
For private Hugging Face models, provide your access token:
```python
deployment = client.deploy_vllm(
name="private-model",
model="your-org/private-model",
env={"HF_TOKEN": "hf_..."},
)
```
Keep your Hugging Face token secure. Consider using environment variables or secrets management.
### Gated Models
Some models require accepting a license agreement on Hugging Face:
1. Visit the model page (e.g., `https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct`)
2. Accept the license agreement
3. Deploy with your token:
```python
deployment = client.deploy_vllm(
name="llama-api",
model="meta-llama/Llama-3.1-8B-Instruct",
env={"HF_TOKEN": "hf_..."},
)
```
## Fine-Tuned Models
### LoRA Adapters
Deploy a base model with LoRA adapter:
```python
# Upload your adapter to Hugging Face first
# huggingface-cli upload your-org/my-adapter ./adapter
deployment = client.deploy_vllm(
name="lora-model",
model="meta-llama/Llama-3.1-8B-Instruct",
env={
"HF_TOKEN": "hf_...",
"VLLM_LORA_MODULES": "my-adapter=your-org/my-adapter",
},
)
```
Use the adapter in requests:
```python
from openai import OpenAI
client = OpenAI(base_url=f"{deployment.url}/v1", api_key="not-needed")
response = client.chat.completions.create(
model="my-adapter", # Use adapter name
messages=[{"role": "user", "content": "Hello!"}],
)
```
### Multiple LoRA Adapters
Serve multiple adapters from one deployment:
```python
deployment = client.deploy_vllm(
name="multi-lora",
model="meta-llama/Llama-3.1-8B-Instruct",
env={
"HF_TOKEN": "hf_...",
"VLLM_LORA_MODULES": "adapter1=org/adapter1,adapter2=org/adapter2",
"VLLM_MAX_LORAS": "4",
},
)
```
### Merged Models
For fine-tuned models merged with base weights:
```python
# Upload merged model to Hugging Face
# huggingface-cli upload your-org/merged-model ./merged_weights
deployment = client.deploy_vllm(
name="merged-model",
model="your-org/merged-model",
env={"HF_TOKEN": "hf_..."},
)
```
## Quantized Models
### AWQ Quantization
Deploy AWQ-quantized models for lower memory usage:
```python
deployment = client.deploy_vllm(
name="qwen-awq",
model="Qwen/Qwen2.5-7B-Instruct-AWQ",
quantization="awq",
gpu_count=1,
)
```
### GPTQ Quantization
```python
deployment = client.deploy_vllm(
name="mistral-gptq",
model="TheBloke/Mistral-7B-Instruct-v0.2-GPTQ",
quantization="gptq",
gpu_count=1,
)
```
### Custom Quantized Models
Upload your quantized model to Hugging Face:
```python
# Quantize locally
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
model.quantize(tokenizer, quant_config={"w_bit": 4, "q_group_size": 128})
model.save_quantized("./quantized-model")
# Upload to Hugging Face
# huggingface-cli upload your-org/llama-awq ./quantized-model
```
Deploy:
```python
deployment = client.deploy_vllm(
name="custom-awq",
model="your-org/llama-awq",
quantization="awq",
)
```
## Custom Docker Images
For complete control, build a custom image:
### Dockerfile
```dockerfile
FROM vllm/vllm-openai:latest
# Pre-download model weights
ENV HF_TOKEN=hf_...
RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('your-org/your-model')"
# Custom dependencies
RUN pip install custom-package
# Custom entrypoint
COPY start.sh /start.sh
ENTRYPOINT ["/start.sh"]
```
### Deploy Custom Image
```python
deployment = client.deploy(
name="custom-vllm",
image="ghcr.io/your-org/custom-vllm:latest",
port=8000,
gpu_count=1,
memory="16Gi",
env={
"MODEL": "your-org/your-model",
},
)
```
## Model Configuration
### Chat Templates
For models without a default chat template:
```python
deployment = client.deploy_vllm(
name="custom-chat",
model="your-org/base-model",
env={
"VLLM_CHAT_TEMPLATE": "/templates/chat.jinja",
},
)
```
Create a custom template in your image or mount it via storage.
### Special Tokens
Configure special tokens for custom models:
```python
deployment = client.deploy_vllm(
name="custom-tokens",
model="your-org/your-model",
env={
# Custom stop tokens
"VLLM_STOP_TOKEN_IDS": "2,32000",
},
)
```
## Embedding Models
Deploy embedding models for semantic search:
```python
deployment = client.deploy_vllm(
name="embeddings",
model="BAAI/bge-large-en-v1.5",
env={
"VLLM_TASK": "embed",
},
)
```
Use embeddings:
```python
from openai import OpenAI
client = OpenAI(base_url=f"{deployment.url}/v1", api_key="not-needed")
response = client.embeddings.create(
model="BAAI/bge-large-en-v1.5",
input=["Hello world", "Goodbye world"],
)
print(response.data[0].embedding[:5]) # First 5 dimensions
```
## Vision-Language Models
Deploy multimodal models:
```python
deployment = client.deploy_vllm(
name="vision-llm",
model="llava-hf/llava-1.5-7b-hf",
gpu_count=1,
memory="16Gi",
)
```
Use with images:
```python
import base64
with open("image.jpg", "rb") as f:
image_data = base64.b64encode(f.read()).decode()
response = client.chat.completions.create(
model="llava-hf/llava-1.5-7b-hf",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}},
],
}
],
)
```
## Testing Your Model
Before production, verify your model works:
```python
from basilica import BasilicaClient
from openai import OpenAI
basilica = BasilicaClient()
# Deploy
deployment = basilica.deploy_vllm(
name="test-model",
model="your-org/your-model",
env={"HF_TOKEN": "hf_..."},
)
deployment.wait_until_ready(timeout=600)
# Test basic completion
client = OpenAI(base_url=f"{deployment.url}/v1", api_key="not-needed")
response = client.chat.completions.create(
model="your-org/your-model",
messages=[{"role": "user", "content": "Hello!"}],
max_tokens=50,
)
print(f"Response: {response.choices[0].message.content}")
print(f"Tokens: {response.usage.total_tokens}")
# Test streaming
stream = client.chat.completions.create(
model="your-org/your-model",
messages=[{"role": "user", "content": "Count to 5"}],
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
# Cleanup
deployment.delete()
```
## Troubleshooting
### Model Not Loading
Check logs for errors:
```python
print(deployment.logs())
```
Common issues:
* **Authentication:** Verify `HF_TOKEN` is correct
* **Memory:** Model may need more GPU VRAM
* **Architecture:** Ensure model architecture is supported
### Wrong Outputs
* Verify chat template matches your model's training format
* Check tokenizer configuration
* Ensure model wasn't corrupted during upload
### Slow Inference
* Enable tensor parallelism for large models
* Use quantization to reduce memory bandwidth
* Check if context length is too large
### Out of Memory
* Reduce `max_model_len`
* Use quantization
* Add more GPUs with tensor parallelism
## Next Steps
* [Deploy with vLLM](/inference/vllm) for production inference
* [Deploy with SGLang](/inference/sglang) for structured generation
* [GPU configuration](/sdk/gpu) for advanced GPU settings
# LLM Inference
# LLM Inference
Summon Llama, Mistral, Qwen, or any Hugging Face model. One function call, one OpenAI-compatible endpoint. Your model, served.
## Why Basilica for LLM Inference?
* **OpenAI-compatible API:** Drop-in replacement for OpenAI API clients
* **Optimized performance:** PagedAttention, continuous batching, tensor parallelism
* **Simple deployment:** One function call deploys a full inference server
* **Cost-effective:** Pay only for GPU time you use
## Supported Frameworks
| Framework | Best For | Performance |
| --------------------------- | -------------------- | --------------------------------- |
| [vLLM](/inference/vllm) | Production workloads | High throughput, low latency |
| [SGLang](/inference/sglang) | Complex pipelines | Structured generation, multi-turn |
## Quick Start
Deploy a model with a single function call:
```python
from basilica import BasilicaClient
client = BasilicaClient()
# Deploy vLLM with Qwen 2.5
deployment = client.deploy_vllm(
name="qwen-api",
model="Qwen/Qwen2.5-7B-Instruct",
gpu_count=1,
)
print(f"OpenAI-compatible API: {deployment.url}/v1")
```
Use with any OpenAI client:
```python
from openai import OpenAI
client = OpenAI(
base_url=f"{deployment.url}/v1",
api_key="not-needed", # Basilica handles auth
)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```
## Choosing a Framework
### Use vLLM When
* You need **maximum throughput** for chat completions
* You want **battle-tested production stability**
* Your workload is **request/response** based
* You need **tensor parallelism** for large models
### Use SGLang When
* You have **complex multi-turn conversations**
* You need **structured generation** (JSON schemas, constrained outputs)
* You want **RadixAttention** for KV cache reuse
* Your pipeline involves **branching logic**
## Dedicated Methods
The SDK provides dedicated methods for LLM inference:
| Method | Description |
| --------------------------------------------- | ------------------------------------ |
| [`client.deploy_vllm()`](/inference/vllm) | Deploy vLLM OpenAI-compatible server |
| [`client.deploy_sglang()`](/inference/sglang) | Deploy SGLang server |
These methods handle:
* Container image selection
* Environment configuration
* Health check configuration
* Optimized resource defaults
## Model Compatibility
Both vLLM and SGLang support most Hugging Face models:
### Chat Models
* `meta-llama/Llama-3.1-8B-Instruct`
* `meta-llama/Llama-3.1-70B-Instruct`
* `mistralai/Mistral-7B-Instruct-v0.3`
* `mistralai/Mixtral-8x7B-Instruct-v0.1`
* `Qwen/Qwen2.5-7B-Instruct`
* `Qwen/Qwen2.5-72B-Instruct`
* `deepseek-ai/DeepSeek-V2-Chat`
### Base Models
* `meta-llama/Llama-3.1-8B`
* `mistralai/Mistral-7B-v0.3`
* `Qwen/Qwen2.5-7B`
### Code Models
* `codellama/CodeLlama-34b-Instruct-hf`
* `deepseek-ai/deepseek-coder-6.7b-instruct`
* `Qwen/Qwen2.5-Coder-7B-Instruct`
Some models require accepting license agreements on Hugging Face. Set `HF_TOKEN` in your environment or pass it via the `env` parameter.
## API Endpoints
Both vLLM and SGLang expose OpenAI-compatible endpoints:
| Endpoint | Description |
| --------------------------- | ------------------------------ |
| `POST /v1/chat/completions` | Chat completions (recommended) |
| `POST /v1/completions` | Text completions |
| `GET /v1/models` | List available models |
| `GET /health` | Health check |
### Chat Completions
```bash
curl -X POST "${DEPLOYMENT_URL}/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is 2+2?"}
],
"temperature": 0.7,
"max_tokens": 100
}'
```
### Streaming
```python
from openai import OpenAI
client = OpenAI(base_url=f"{deployment.url}/v1", api_key="not-needed")
stream = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
## Next Steps
* **[Deploy with vLLM](/inference/vllm):** Production-grade inference
* **[Deploy with SGLang](/inference/sglang):** Structured generation
* **[Deploy custom models](/inference/custom-models):** Fine-tuned and private models
# SGLang
# SGLang
For when you need structured outputs and complex pipelines. RadixAttention, constrained generation, multi-turn efficiency. Deploy with `deploy_sglang()`.
## Quick Start
Deploy an SGLang inference server with one function call:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy_sglang(
name="qwen-sglang",
model="Qwen/Qwen2.5-7B-Instruct",
gpu_count=1,
)
print(f"API endpoint: {deployment.url}/v1")
```
## API Reference
```python
deployment = client.deploy_sglang(
name: str, # Deployment name
model: str, # Hugging Face model ID
gpu_count: int = 1, # Number of GPUs
tensor_parallel_size: int = None, # Tensor parallelism (default: gpu_count)
context_length: int = None, # Maximum context length
quantization: str = None, # Quantization method
gpu_models: List[str] = None, # GPU model filter
min_gpu_memory_gb: int = None, # Minimum GPU VRAM
memory: str = "16Gi", # CPU memory allocation
env: Dict[str, str] = None, # Additional environment variables
timeout: int = 600, # Deployment timeout
) -> Deployment
```
### Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ---------------- | ------------- | ------------------------------------- |
| `name` | `str` | required | Deployment name (DNS-safe) |
| `model` | `str` | required | Hugging Face model ID |
| `gpu_count` | `int` | `1` | Number of GPUs to allocate |
| `tensor_parallel_size` | `int` | `gpu_count` | Number of GPUs for tensor parallelism |
| `context_length` | `int` | model default | Maximum context length |
| `quantization` | `str` | `None` | Quantization method |
| `gpu_models` | `List[str]` | `None` | Acceptable GPU models |
| `min_gpu_memory_gb` | `int` | `None` | Minimum GPU VRAM in GB |
| `memory` | `str` | `"16Gi"` | CPU memory allocation |
| `env` | `Dict[str, str]` | `None` | Additional environment variables |
| `timeout` | `int` | `600` | Seconds to wait for ready |
## Why SGLang?
### RadixAttention
SGLang's RadixAttention efficiently reuses KV cache across requests with shared prefixes. This is beneficial for:
* **System prompts:** Shared system message across conversations
* **Few-shot learning:** Reuse examples across queries
* **Document QA:** Multiple questions about the same document
### Structured Generation
SGLang excels at generating structured outputs:
* JSON with schema validation
* Regex-constrained outputs
* Choice selection from options
## Basic Examples
### Deploy Qwen 2.5
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy_sglang(
name="qwen-sglang",
model="Qwen/Qwen2.5-7B-Instruct",
)
```
### Deploy Llama 3.1
```python
deployment = client.deploy_sglang(
name="llama-sglang",
model="meta-llama/Llama-3.1-8B-Instruct",
env={"HF_TOKEN": "hf_..."},
)
```
### Deploy Mistral
```python
deployment = client.deploy_sglang(
name="mistral-sglang",
model="mistralai/Mistral-7B-Instruct-v0.3",
)
```
## Using the API
SGLang provides an OpenAI-compatible API:
### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
base_url=f"{deployment.url}/v1",
api_key="not-needed",
)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain machine learning in simple terms."},
],
temperature=0.7,
max_tokens=500,
)
print(response.choices[0].message.content)
```
### Streaming
```python
stream = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Write a haiku about programming"}],
stream=True,
)
for chunk in stream:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)
```
### cURL
```bash
curl -X POST "${DEPLOYMENT_URL}/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
## Structured Generation
SGLang's key feature is structured generation. Use the native API for advanced features:
### JSON Output
```python
import httpx
response = httpx.post(
f"{deployment.url}/generate",
json={
"text": "Generate a user profile:",
"sampling_params": {
"temperature": 0.7,
"max_new_tokens": 200,
},
"return_logprob": False,
"regex": r'\{"name": "[^"]+", "age": \d+, "email": "[^"]+"\}',
},
)
print(response.json()["text"])
# {"name": "John Smith", "age": 28, "email": "john@example.com"}
```
### Choice Selection
```python
response = httpx.post(
f"{deployment.url}/generate",
json={
"text": "Is Python a compiled or interpreted language?",
"sampling_params": {"temperature": 0},
"choices": ["compiled", "interpreted"],
},
)
print(response.json()["text"]) # "interpreted"
```
### JSON Schema
```python
schema = {
"type": "object",
"properties": {
"sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"keywords": {"type": "array", "items": {"type": "string"}},
},
"required": ["sentiment", "confidence", "keywords"],
}
response = httpx.post(
f"{deployment.url}/generate",
json={
"text": "Analyze the sentiment: 'I love this product! It works great.'",
"sampling_params": {"temperature": 0.1, "max_new_tokens": 100},
"json_schema": schema,
},
)
import json
result = json.loads(response.json()["text"])
print(result)
# {"sentiment": "positive", "confidence": 0.95, "keywords": ["love", "great", "works"]}
```
## Advanced Configuration
### Tensor Parallelism
For large models:
```python
deployment = client.deploy_sglang(
name="llama-70b-sglang",
model="meta-llama/Llama-3.1-70B-Instruct",
gpu_count=4,
tensor_parallel_size=4,
memory="128Gi",
timeout=900,
)
```
### Custom Context Length
```python
deployment = client.deploy_sglang(
name="long-context",
model="Qwen/Qwen2.5-7B-Instruct",
context_length=32768,
min_gpu_memory_gb=24,
)
```
### Performance Tuning
```python
deployment = client.deploy_sglang(
name="optimized-sglang",
model="Qwen/Qwen2.5-7B-Instruct",
env={
# Memory management
"SGL_MEM_FRACTION_STATIC": "0.85",
# Chunked prefill for better throughput
"SGL_CHUNKED_PREFILL_SIZE": "8192",
# Attention backend
"SGL_ATTENTION_BACKEND": "flashinfer",
},
)
```
## SGLang Native API
Beyond OpenAI compatibility, SGLang provides a native API with additional features:
### Generate Endpoint
```python
import httpx
response = httpx.post(
f"{deployment.url}/generate",
json={
"text": "The capital of France is",
"sampling_params": {
"temperature": 0,
"max_new_tokens": 50,
},
},
)
print(response.json()["text"])
```
### Batch Generation
```python
response = httpx.post(
f"{deployment.url}/generate",
json={
"text": [
"Translate to French: Hello",
"Translate to French: Goodbye",
"Translate to French: Thank you",
],
"sampling_params": {"temperature": 0, "max_new_tokens": 50},
},
)
for result in response.json():
print(result["text"])
```
### Get Model Info
```python
response = httpx.get(f"{deployment.url}/get_model_info")
print(response.json())
```
## Gated Models
For models requiring Hugging Face authentication:
```python
deployment = client.deploy_sglang(
name="llama-sglang",
model="meta-llama/Llama-3.1-8B-Instruct",
env={"HF_TOKEN": "hf_..."},
)
```
See [Custom Models](/inference/custom-models#gated-models) for detailed setup.
## Monitoring
### Check Status
```python
status = deployment.status()
print(f"State: {status.state}")
print(f"Ready: {status.is_ready}")
```
### View Logs
```python
print(deployment.logs(tail=50))
```
### Health Check
```python
import httpx
response = httpx.get(f"{deployment.url}/health")
print(response.json())
```
## API Endpoints
| Endpoint | Method | Description |
| ---------------------- | ------ | --------------------------------- |
| `/v1/chat/completions` | POST | OpenAI-compatible chat |
| `/v1/completions` | POST | OpenAI-compatible completion |
| `/v1/models` | GET | List models |
| `/generate` | POST | Native SGLang generation |
| `/get_model_info` | GET | Model information |
| `/health` | GET | Health check |
| `/health_generate` | GET | Health check with generation test |
## SGLang vs vLLM
| Feature | SGLang | vLLM |
| --------------------- | ------ | --------- |
| OpenAI API | Yes | Yes |
| Throughput | High | Very High |
| RadixAttention | Yes | No |
| Structured Generation | Native | Limited |
| JSON Schema | Yes | No |
| Regex Constraints | Yes | No |
| Choice Selection | Yes | No |
| Multi-turn Efficiency | Better | Good |
**Choose SGLang when:**
* You need structured outputs (JSON, regex)
* Your workload has shared prefixes (system prompts, documents)
* You're building complex multi-turn conversations
**Choose vLLM when:**
* Raw throughput is the priority
* You have simple request/response patterns
* You need maximum battle-tested stability
## Troubleshooting
### Model Won't Load
```python
print(deployment.logs())
```
Common issues:
* **Out of memory:** Reduce context length or add GPUs
* **Missing token:** Set `HF_TOKEN` for gated models
### Generation Errors
For structured generation issues:
* Ensure schema is valid JSON Schema
* Check regex patterns are correct
* Verify model supports the output format
### Slow Performance
* Enable tensor parallelism for large models
* Use `SGL_CHUNKED_PREFILL_SIZE` for better batching
* Reduce context length if not needed
## Next Steps
* [Deploy with vLLM](/inference/vllm) for maximum throughput
* [Deploy custom models](/inference/custom-models) for fine-tuned models
* [GPU configuration](/sdk/gpu) for advanced GPU settings
# vLLM
# vLLM
The gold standard for LLM inference. PagedAttention, continuous batching, blazing throughput. Deploy with `deploy_vllm()`.
## Quick Start
Deploy a vLLM inference server with one function call:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy_vllm(
name="qwen-api",
model="Qwen/Qwen2.5-7B-Instruct",
gpu_count=1,
)
print(f"API endpoint: {deployment.url}/v1")
```
## API Reference
```python
deployment = client.deploy_vllm(
name: str, # Deployment name
model: str, # Hugging Face model ID
gpu_count: int = 1, # Number of GPUs
tensor_parallel_size: int = None, # Tensor parallelism (default: gpu_count)
max_model_len: int = None, # Maximum sequence length
quantization: str = None, # Quantization method
gpu_models: List[str] = None, # GPU model filter
min_gpu_memory_gb: int = None, # Minimum GPU VRAM
memory: str = "16Gi", # CPU memory allocation
env: Dict[str, str] = None, # Additional environment variables
timeout: int = 600, # Deployment timeout
) -> Deployment
```
### Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ---------------- | ------------- | ---------------------------------------------------------- |
| `name` | `str` | required | Deployment name (DNS-safe) |
| `model` | `str` | required | Hugging Face model ID (e.g., `"Qwen/Qwen2.5-7B-Instruct"`) |
| `gpu_count` | `int` | `1` | Number of GPUs to allocate |
| `tensor_parallel_size` | `int` | `gpu_count` | Number of GPUs for tensor parallelism |
| `max_model_len` | `int` | model default | Maximum context length |
| `quantization` | `str` | `None` | Quantization: `"awq"`, `"gptq"`, `"squeezellm"` |
| `gpu_models` | `List[str]` | `None` | Acceptable GPU models |
| `min_gpu_memory_gb` | `int` | `None` | Minimum GPU VRAM in GB |
| `memory` | `str` | `"16Gi"` | CPU memory allocation |
| `env` | `Dict[str, str]` | `None` | Additional environment variables |
| `timeout` | `int` | `600` | Seconds to wait for ready |
## Basic Examples
### Deploy Qwen 2.5
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy_vllm(
name="qwen-7b",
model="Qwen/Qwen2.5-7B-Instruct",
)
```
### Deploy Llama 3.1
```python
deployment = client.deploy_vllm(
name="llama-8b",
model="meta-llama/Llama-3.1-8B-Instruct",
env={"HF_TOKEN": "hf_..."}, # Llama requires license acceptance
)
```
### Deploy Mistral
```python
deployment = client.deploy_vllm(
name="mistral-7b",
model="mistralai/Mistral-7B-Instruct-v0.3",
)
```
## Using the API
vLLM exposes an OpenAI-compatible API. Use any OpenAI client:
### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
base_url=f"{deployment.url}/v1",
api_key="not-needed",
)
# Chat completion
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in simple terms."},
],
temperature=0.7,
max_tokens=500,
)
print(response.choices[0].message.content)
```
### Streaming Responses
```python
stream = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Write a short poem about AI"}],
stream=True,
)
for chunk in stream:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)
```
### cURL
```bash
curl -X POST "${DEPLOYMENT_URL}/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
### JavaScript/TypeScript
```typescript
import OpenAI from 'openai';
const client = new OpenAI({
baseURL: `${deploymentUrl}/v1`,
apiKey: 'not-needed',
});
const response = await client.chat.completions.create({
model: 'Qwen/Qwen2.5-7B-Instruct',
messages: [{ role: 'user', content: 'Hello!' }],
});
console.log(response.choices[0].message.content);
```
## Advanced Configuration
### Tensor Parallelism
For large models that don't fit on a single GPU:
```python
# 70B model across 4 GPUs
deployment = client.deploy_vllm(
name="llama-70b",
model="meta-llama/Llama-3.1-70B-Instruct",
gpu_count=4,
tensor_parallel_size=4,
memory="128Gi",
timeout=900, # Large models take longer to load
)
```
### Custom Context Length
```python
# Shorter context for faster inference
deployment = client.deploy_vllm(
name="fast-qwen",
model="Qwen/Qwen2.5-7B-Instruct",
max_model_len=4096, # Default is 32k
)
# Extended context (requires more VRAM)
deployment = client.deploy_vllm(
name="long-context",
model="Qwen/Qwen2.5-7B-Instruct",
max_model_len=32768,
min_gpu_memory_gb=24,
)
```
### Quantized Models
Deploy AWQ or GPTQ quantized models for lower memory usage:
```python
deployment = client.deploy_vllm(
name="qwen-awq",
model="Qwen/Qwen2.5-7B-Instruct-AWQ",
quantization="awq",
)
```
See [Custom Models](/inference/custom-models#quantized-models) for GPTQ examples and custom quantization.
### Performance Tuning
```python
deployment = client.deploy_vllm(
name="high-throughput",
model="Qwen/Qwen2.5-7B-Instruct",
env={
# Increase batch size for throughput
"VLLM_MAX_NUM_BATCHED_TOKENS": "8192",
"VLLM_MAX_NUM_SEQS": "256",
# Enable eager mode (can help with some models)
"VLLM_ENFORCE_EAGER": "true",
# Swap space for larger batches
"VLLM_SWAP_SPACE": "4", # GB
},
)
```
### Specific GPU Selection
```python
deployment = client.deploy_vllm(
name="a100-inference",
model="meta-llama/Llama-3.1-70B-Instruct",
gpu_count=2,
gpu_models=["A100"],
min_gpu_memory_gb=80,
)
```
## Gated Models
For models requiring Hugging Face license acceptance:
```python
deployment = client.deploy_vllm(
name="llama-api",
model="meta-llama/Llama-3.1-8B-Instruct",
env={"HF_TOKEN": "hf_..."},
)
```
See [Custom Models](/inference/custom-models#gated-models) for detailed setup instructions.
## Monitoring
### Check Status
```python
status = deployment.status()
print(f"State: {status.state}")
print(f"Phase: {status.phase}")
print(f"Ready: {status.is_ready}")
```
### View Logs
```python
# Recent logs
print(deployment.logs(tail=50))
# All logs
print(deployment.logs())
```
### Health Check
```python
import httpx
response = httpx.get(f"{deployment.url}/health")
print(response.json()) # {"status": "healthy"}
```
## API Endpoints
| Endpoint | Method | Description |
| ---------------------- | ------ | ------------------- |
| `/v1/chat/completions` | POST | Chat completions |
| `/v1/completions` | POST | Text completions |
| `/v1/models` | GET | List models |
| `/v1/embeddings` | POST | Generate embeddings |
| `/health` | GET | Health check |
| `/version` | GET | vLLM version |
## Supported Parameters
Chat completions support these parameters:
| Parameter | Type | Default | Description |
| ------------------- | ------- | -------- | -------------------------- |
| `model` | `str` | required | Model identifier |
| `messages` | `list` | required | Conversation messages |
| `temperature` | `float` | `1.0` | Sampling temperature |
| `top_p` | `float` | `1.0` | Nucleus sampling |
| `top_k` | `int` | `-1` | Top-k sampling |
| `max_tokens` | `int` | `16` | Maximum tokens to generate |
| `stream` | `bool` | `false` | Enable streaming |
| `stop` | `list` | `None` | Stop sequences |
| `presence_penalty` | `float` | `0.0` | Presence penalty |
| `frequency_penalty` | `float` | `0.0` | Frequency penalty |
| `n` | `int` | `1` | Number of completions |
## Troubleshooting
### Model Won't Load
Check logs for errors:
```python
print(deployment.logs())
```
Common causes:
* **Out of memory:** Use smaller `max_model_len` or more GPUs
* **Missing token:** Gated models need `HF_TOKEN`
* **Wrong quantization:** Match `quantization` to model type
### Slow Startup
Large models take time to download and load:
```python
deployment = client.deploy_vllm(
name="large-model",
model="meta-llama/Llama-3.1-70B-Instruct",
timeout=1200, # 20 minutes
)
```
### High Latency
* Reduce `max_model_len` for shorter context
* Enable tensor parallelism with more GPUs
* Use quantized models for faster inference
## Next Steps
* [Deploy with SGLang](/inference/sglang) for structured generation
* [Deploy custom models](/inference/custom-models) for fine-tuned models
* [GPU configuration](/sdk/gpu) for advanced GPU settings
# Current Configuration
# Current Configuration
These values can change. This page reflects what's live in production today.
| Parameter | Value |
| -------------------------- | ----------------- |
| **Default vesting window** | 72 hours (3 days) |
| **Revenue share** | 50% |
## Per-GPU configuration
| GPU Type | Rate (per GPU per hour) | Target | Vesting Window |
| -------- | ----------------------- | ------ | -------------- |
| A100 | $1.50 | 8 GPUs | default (72h) |
| H100 | $2.00 | 8 GPUs | default (72h) |
## How these values drive earnings
* **Rate** is your CU payout per GPU per hour of validated uptime.
* **Target** sets the dilution threshold. When online supply exceeds the target for a GPU type, per-GPU CU earnings decrease proportionally. At or below target, you earn the full rate.
* **Revenue share** determines what percentage of rental revenue flows to you as RU payouts, on top of CU earnings.
For vesting curves, dilution math, and worked examples, see [Emissions](/miners/emissions).
# Emissions
# Emissions
Understand how the network rewards your faithful compute.
## How do you earn?
Your payout comes from two independent streams:
```
Miner Payout = CU Payout + RU Payout
```
1. **Compute Units (CUs)**: 1 CU = 1 GPU-hour of validated availability. You earn CUs by keeping your node online and passing validation, whether or not it gets rented. Each CU is priced at a configured rate per GPU-hour.
2. **Revenue Units (RUs)**: actual USD rental revenue when your node is rented. A configurable share of this revenue is paid on top of CU earnings.
All rates are **per GPU per hour**. A node with 8 GPUs earns 8x the listed rate.
## Vesting
Both CUs and RUs vest linearly over a configurable window (3 days by default). At each weight-setting epoch, validators calculate how much has vested and set your on-chain weight proportionally.
Going offline only stops you from earning **new** CUs. Already-earned CUs and RUs continue vesting normally.
## Dilution
Each GPU type has a **target supply** for the network. When more GPUs of a type come online than the target, the CU budget splits across all online GPUs. Each one earns proportionally less.
* **At or below target**: you earn the full configured rate (no bonus for undersupply).
* **Above target**: the budget splits evenly, reducing per-GPU CU earnings.
* **Categories are independent**: oversupply of H100s does not affect A100 CU payouts.
*Example: if the target for a GPU type is 24 GPUs and 48 are online, the per-GPU CU rate is halved.*
RU payouts are not affected by dilution. They come directly from your actual rental revenue.
## Slashing
Slashing only happens when your node is **permanently lost during an active rental**. This means the validator has given up health-checking the node. When slashed, a portion of your unvested CUs and RUs are voided.
The following do **not** trigger slashing:
* Transient failures
* Container health issues
* User-initiated rental stops
Recovery is automatic. New CUs accrue normally once the node comes back online.
## See it in action
Using example values of $2.00/hr per GPU and 50% revenue share:
**1 node, 8x GPUs, online 24/7, supply at target:**
* **CU earnings**: 8 GPUs × $2.00/hr × 24h = **$384/day** in emission value
* **RU earnings** (if rented 12h at a $3.00/hr bid): revenue = 8 × $3.00 × 12 = $288, your 50% share = **$144/day extra**
* Total: **$528/day**, vesting linearly over the configured window
These are illustrative values. Refer to [Current Configuration](/miners/configuration) for actual rates.
# Mining on Basilica
# Mining on Basilica
Earn emissions by running GPU nodes on the Basilica network.
## Quick Start
Already have your GPU nodes prepared? Here's the short version:
```bash
# 1. Generate SSH key for node access
ssh-keygen -t ed25519 -f ~/.ssh/miner_node_key -N ""
# 2. Deploy key to GPU nodes
ssh-copy-id -i ~/.ssh/miner_node_key.pub basilica@
# 3. Create config
cat > miner.toml < Validator
Miner ──Deploy SSH Key──> GPU Nodes
Validator ──SSH──> GPU Nodes (validation + rentals)
Miner ──HealthCheck──> Validator (periodic)
```
## Next Steps
* **[Requirements](/miners/requirements):** Prepare your GPU nodes
* **[Setup](/miners/setup):** Configure and deploy the miner
* **[Emissions](/miners/emissions):** Understand how you earn
# Requirements
# Requirements
Get your miner server and GPU nodes ready before deploying.
## Miner Server
A Linux machine that orchestrates access to your GPU nodes. CPU only, no GPU required.
* Public IP address (or port forwarding for axon port)
* SSH access to all your GPU nodes
**If building from source:**
```bash
sudo apt update && sudo apt install -y \
build-essential libssl-dev pkg-config protobuf-compiler git curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
**If deploying via Docker/Docker Compose:**
```bash
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
```
## GPU Nodes
Your GPU nodes are your offering to the network. They must be **bare metal or VMs**, not containers. We need full access to the machine and its networking stack. Docker-in-Docker and SSH inside containers are not supported.
Each GPU node needs the following.
### Supported GPUs
A100, H100, H200, B200. Check the [current configuration](/miners/configuration) to confirm your GPU type is earning emissions.
### NVIDIA Drivers + CUDA
CUDA 12.8 or higher. Verify with:
```bash
nvidia-smi
```
### Docker + NVIDIA Container Toolkit
```bash
# Install Docker
curl -fsSL https://get.docker.com | sudo sh
# Install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Verify GPU access in Docker
docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi
```
### Set up a dedicated user
Create a `basilica` user with Docker access:
```bash
sudo useradd -m -s /bin/bash basilica
sudo usermod -aG docker basilica
# Set up SSH
sudo -u basilica mkdir -p /home/basilica/.ssh
sudo chmod 700 /home/basilica/.ssh
sudo -u basilica touch /home/basilica/.ssh/authorized_keys
sudo chmod 600 /home/basilica/.ssh/authorized_keys
```
Verify the user can access GPUs:
```bash
sudo -u basilica docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi
```
### Network
* SSH server running and accessible from the miner server and the validator
* Firewall allows inbound SSH from both miner and validator IPs
* The validator controls which additional ports need open internet access
```bash
# Allow SSH (UFW)
sudo ufw allow 22/tcp
```
### Disk space
At least **1TB of free disk space**. This can be on the root mount or a dedicated NVMe/SSD mount.
If your storage is on a separate device, mount it on the host and declare it in your [node config](/miners/setup#define-your-nodes). The validator will make it available to rental containers.
## Register your Bittensor wallet
Register on subnet 39 (mainnet):
```bash
pip install bittensor
# Create wallet
btcli wallet new_coldkey --wallet.name miner_wallet
btcli wallet new_hotkey --wallet.name miner_wallet --wallet.hotkey default
# Register on mainnet
btcli subnet register --netuid 39 --wallet.name miner_wallet --wallet.hotkey default
```
## Before you proceed
Verify each GPU node has:
* [ ] NVIDIA drivers installed (`nvidia-smi` works)
* [ ] Docker with NVIDIA runtime (`docker run --gpus all` works)
* [ ] Dedicated `basilica` user in `docker` group
* [ ] SSH server running and accessible
* [ ] 1TB+ free disk space (root mount or dedicated storage mount)
* [ ] Firewall allows SSH from miner + validator
## Next Steps
* **[Setup](/miners/setup):** Configure SSH keys, miner.toml, and deploy
# Setup
# Setup
Configure your SSH keys, set up the miner, and deploy.
## Generate SSH keys
Your miner needs an SSH key to reach your GPU nodes. This key grants passage to each machine.
```bash
ssh-keygen -t ed25519 -f ~/.ssh/miner_node_key -C "basilica-miner" -N ""
chmod 600 ~/.ssh/miner_node_key
```
Deploy to each GPU node:
```bash
ssh-copy-id -i ~/.ssh/miner_node_key.pub basilica@
```
Verify:
```bash
ssh -i ~/.ssh/miner_node_key basilica@ "hostname && nvidia-smi --query-gpu=name --format=csv,noheader"
```
## Configure the miner
Clone the Basilica repo and copy the config template:
```bash
git clone https://github.com/one-covenant/basilica.git
mkdir -p config
cp basilica/config/miner.toml.example config/miner.toml
```
Edit `config/miner.toml` with your settings:
### Bittensor
```toml
[bittensor]
wallet_name = "miner_wallet"
hotkey_name = "default"
external_ip = "" # curl -4 ifconfig.me
axon_port = 50051
network = "finney" # Mainnet
netuid = 39 # Mainnet subnet
chain_endpoint = "wss://entrypoint-finney.opentensor.ai:443"
```
Open the axon port on your firewall:
```bash
sudo ufw allow 50051/tcp
```
### Database
```toml
[database]
url = "sqlite:///opt/basilica/data/miner.db"
run_migrations = true
```
```bash
sudo mkdir -p /opt/basilica/data && sudo chown $USER:$USER /opt/basilica/data
```
### Define your nodes
List your GPU nodes below. Every GPU category listed here must have a matching price in the bidding section.
```toml
[node_management]
nodes = [
{ host = "192.168.1.100", port = 22, username = "basilica", gpu_category = "H100", gpu_count = 8 },
{ host = "192.168.1.101", port = 22, username = "basilica", gpu_category = "A100", gpu_count = 8, extra_mount_path = "/ephemeral" },
]
```
`host` must be an IPv4 address, not a hostname.
`extra_mount_path` is optional. Set it if your node has NVMe/SSD storage mounted at a dedicated path (e.g., `/ephemeral`). The storage will be made available to rental containers.
### Set your prices
Price each GPU type per hour in dollars. Every category in your node list must have an entry here.
```toml
[bidding.strategy.static.static_prices]
H100 = 2.50 # $2.50/hr per GPU
A100 = 1.20 # $1.20/hr per GPU
```
The validator enforces a minimum bid floor (default 10% of baseline market price). Bids below this are rejected.
### SSH
```toml
[ssh_session]
miner_node_key_path = "~/.ssh/miner_node_key"
default_node_username = "basilica"
```
### Choose a validator
```toml
[validator_assignment]
strategy = "highest_stake" # Recommended: auto-selects highest-stake validator
```
Use `fixed_assignment` with a `validator_hotkey` only for debugging.
### Metrics (Optional)
```toml
[metrics]
enabled = true
[metrics.prometheus]
host = "127.0.0.1"
port = 9090
```
## Deploy
### Docker Compose (Recommended)
Clone the Basilica repo and use the production compose file:
```bash
# If you haven't already cloned the repo during configuration:
git clone https://github.com/one-covenant/basilica.git
cp basilica/scripts/miner/compose.prod.yml compose.yml
docker compose up -d
```
Watchtower handles automatic image updates. Check status:
```bash
docker compose ps
docker compose logs -f miner
```
### Build from source
```bash
# Build
./scripts/miner/build.sh
# Run
sudo mkdir -p /opt/basilica/{data,config}
sudo cp miner.toml /opt/basilica/config/
./basilica-miner --config /opt/basilica/config/miner.toml
```
## Verify it's working
After the miner starts, look for these messages in the logs:
* `Node registered`: your nodes were accepted by the validator
* `Validator authenticated`: the validator's SSH key was deployed
```bash
# Docker Compose
docker compose logs -f miner
# Binary
# check terminal output directly
# Metrics
curl http://localhost:9090/metrics
```
## Troubleshooting
### Miner won't start
* Verify wallet exists: `ls ~/.bittensor/wallets//hotkeys/`
* Check config: `./basilica-miner --config miner.toml config validate`
* Ensure every GPU category in nodes has a matching price in `static_prices`
### Nodes not registering
* Verify SSH access: `ssh -i ~/.ssh/miner_node_key basilica@`
* Check that `external_ip` is your actual public IP
* Ensure axon port (50051) is open
### Validator rejects bids
Your bid price may be below the minimum floor. Check logs for the required minimum and adjust `static_prices`.
### How bans work
Validators track node misbehaviour and can temporarily ban nodes that repeatedly fail in ways that affect rentals.
A ban triggers when a node accumulates `3+` misbehaviour events within any rolling 24-hour window.
| Offences Recorded In The Last 7 Days | Ban Duration |
| ------------------------------------ | ------------ |
| 1 | 1 hour |
| 2 | 2 hours |
| 3 | 4 hours |
| 4 | 8 hours |
| 5+ | 24 hours |
In practice, the first ban that actually activates is usually 4 hours, because bans do not trigger until the third qualifying misbehaviour.
Misbehaviour events currently include:
* Deployment failures during rental startup
* Interrupted or halted rentals detected by validator health checks
* GPU declaration mismatches detected during full validation
Bans clear automatically. Fix the root cause (drivers, reachability, Docker) and wait for expiry.
## Next Steps
* **[Emissions](/miners/emissions):** Understand how you earn
# Deployments
# Deployments
## Using the Decorator
```python
import basilica
@basilica.deployment(name="hello", port=8000)
def serve():
from http.server import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
self.wfile.write(b'Hello!')
HTTPServer(('', 8000), Handler).serve_forever()
deployment = serve()
print(f"Live at: {deployment.url}")
```
### Parameters
| Parameter | Type | Default | Description |
| -------------- | ---------------- | -------------------- | -------------------------- |
| `name` | `str` | required | Deployment name (DNS-safe) |
| `image` | `str` | `"python:3.11-slim"` | Container image |
| `port` | `int` | `8000` | Port your app listens on |
| `cpu` | `str` | `"500m"` | CPU allocation |
| `memory` | `str` | `"512Mi"` | Memory allocation |
| `env` | `Dict[str, str]` | `None` | Environment variables |
| `pip_packages` | `List[str]` | `None` | Pip dependencies |
| `replicas` | `int` | `1` | Number of instances |
| `ttl_seconds` | `int` | `None` | Auto-delete timer |
| `timeout` | `int` | `300` | Deploy timeout (seconds) |
For GPU options, see [GPU Workloads](/sdk/gpu). For persistent storage, see [Storage](/sdk/storage).
### DeployedFunction API
```python
serve.spec # DeploymentSpec configuration
serve.deployment # Current Deployment (after deploy)
serve() # Deploy (shorthand for .deploy())
serve.deploy() # Deploy with default client
serve.local() # Run locally for testing
```
### Local Testing
```python
@basilica.deployment(name="app", port=8000)
def serve():
print("Starting server...")
serve.local() # Runs locally, doesn't deploy
```
## Using client.deploy()
Use for existing files, pre-built images, or dynamic scenarios.
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="my-app",
source="app.py",
port=8000,
pip_packages=["fastapi", "uvicorn"],
)
```
### Source Options
```python
# File path
client.deploy(name="app", source="app.py", port=8000)
# Inline code
client.deploy(name="app", source="print('hello')", port=8000)
# Image only (no source)
client.deploy(name="nginx", image="nginx:alpine", port=80)
```
## Deployment Object
Every deployment returns a `Deployment` object:
| Property | Type | Description |
| ------------ | ---------- | -------------------- |
| `url` | `str` | Public URL |
| `name` | `str` | Deployment name |
| `state` | `str` | Current state |
| `namespace` | `str` | Kubernetes namespace |
| `user_id` | `str` | Owner's user ID |
| `created_at` | `datetime` | Creation timestamp |
### Methods
```python
deployment.status() # Get detailed status
deployment.logs() # Get container logs
deployment.logs(tail=50) # Last 50 lines
deployment.delete() # Delete deployment
deployment.wait_until_ready() # Wait for ready state
```
### Status Details
```python
status = deployment.status()
print(f"State: {status.state}")
print(f"Ready: {status.replicas_ready}/{status.replicas_desired}")
```
Status properties: `state`, `phase`, `replicas_ready`, `replicas_desired`, `is_ready`, `is_failed`, `is_pending`.
## Management
```python
# Get deployment by name
deployment = client.get("my-app")
# List all deployments
for d in client.list():
print(f"{d.name}: {d.state}")
# Delete
deployment.delete()
```
## Next Steps
* [Storage](/sdk/storage) - Persist data across restarts
* [GPU Workloads](/sdk/gpu) - Deploy on GPU hardware
* [Examples](/sdk/examples) - More deployment patterns
# Examples
# Examples
Copy, paste, deploy. Ready-to-use patterns for common scenarios.
## Basic Examples
### Hello World
The simplest possible deployment:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="hello",
source="""
from http.server import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
self.wfile.write(b'Hello World!')
HTTPServer(('', 8000), Handler).serve_forever()
""",
port=8000,
)
print(f"Live at: {deployment.url}")
```
### Static File Server
Serve static files with nginx:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="static",
image="nginxinc/nginx-unprivileged:alpine",
port=8080,
)
print(f"Nginx at: {deployment.url}")
```
## Web Frameworks
### FastAPI
```python
import basilica
@basilica.deployment(
name="fastapi-app",
port=8000,
pip_packages=["fastapi", "uvicorn"],
)
def serve():
from fastapi import FastAPI
import uvicorn
app = FastAPI(title="My API")
@app.get("/")
def root():
return {"message": "Hello from FastAPI!"}
@app.get("/items/{item_id}")
def get_item(item_id: int, q: str = None):
return {"item_id": item_id, "q": q}
@app.post("/items")
def create_item(item: dict):
return {"created": item}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
print(f"API docs: {deployment.url}/docs")
```
### Flask
```python
import basilica
@basilica.deployment(
name="flask-app",
port=5000,
pip_packages=["flask", "gunicorn"],
)
def serve():
from flask import Flask, jsonify
import subprocess
app = Flask(__name__)
@app.route("/")
def home():
return jsonify({"message": "Hello from Flask!"})
@app.route("/health")
def health():
return jsonify({"status": "healthy"})
# Run with gunicorn for production
subprocess.run([
"gunicorn",
"--bind", "0.0.0.0:5000",
"--workers", "2",
"app:app"
])
deployment = serve()
```
### Streamlit
```python
import basilica
@basilica.deployment(
name="streamlit-app",
port=8501,
pip_packages=["streamlit"],
)
def serve():
import subprocess
import textwrap
from pathlib import Path
# Write Streamlit app
app_code = textwrap.dedent("""
import streamlit as st
st.title("Hello Streamlit!")
st.write("This app is running on Basilica.")
name = st.text_input("Your name")
if name:
st.write(f"Hello, {name}!")
number = st.slider("Pick a number", 0, 100, 50)
st.write(f"You picked: {number}")
""")
Path("/tmp/app.py").write_text(app_code)
subprocess.run([
"streamlit", "run", "/tmp/app.py",
"--server.port", "8501",
"--server.address", "0.0.0.0",
])
deployment = serve()
```
### Gradio
```python
import basilica
@basilica.deployment(
name="gradio-app",
port=7860,
pip_packages=["gradio"],
)
def serve():
import gradio as gr
def greet(name):
return f"Hello, {name}!"
demo = gr.Interface(
fn=greet,
inputs="text",
outputs="text",
title="Greeting App",
)
demo.launch(server_name="0.0.0.0", server_port=7860)
deployment = serve()
```
## Storage Examples
### Persistent Counter
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="counter",
source="""
from pathlib import Path
from http.server import HTTPServer, BaseHTTPRequestHandler
COUNTER_FILE = Path('/data/count.txt')
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
count = int(COUNTER_FILE.read_text()) if COUNTER_FILE.exists() else 0
count += 1
COUNTER_FILE.write_text(str(count))
self.send_response(200)
self.end_headers()
self.wfile.write(f'Visit #{count}'.encode())
HTTPServer(('', 8000), Handler).serve_forever()
""",
port=8000,
storage=True,
)
```
### File Upload API
```python
import basilica
uploads = basilica.Volume.from_name("uploads", create_if_missing=True)
@basilica.deployment(
name="upload-api",
port=8000,
pip_packages=["fastapi", "uvicorn", "python-multipart"],
volumes={"/uploads": uploads},
)
def serve():
from fastapi import FastAPI, UploadFile, HTTPException
from fastapi.responses import FileResponse
from pathlib import Path
import uvicorn
app = FastAPI()
UPLOAD_DIR = Path("/uploads")
@app.post("/upload")
async def upload(file: UploadFile):
path = UPLOAD_DIR / file.filename
with open(path, "wb") as f:
content = await file.read()
f.write(content)
return {"filename": file.filename, "size": len(content)}
@app.get("/files")
def list_files():
files = [f.name for f in UPLOAD_DIR.iterdir() if f.is_file()]
return {"files": files}
@app.get("/files/{filename}")
def download(filename: str):
path = UPLOAD_DIR / filename
if not path.exists():
raise HTTPException(404, "File not found")
return FileResponse(path)
@app.delete("/files/{filename}")
def delete(filename: str):
path = UPLOAD_DIR / filename
if not path.exists():
raise HTTPException(404, "File not found")
path.unlink()
return {"deleted": filename}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
### SQLite Database
```python
import basilica
data = basilica.Volume.from_name("app-db", create_if_missing=True)
@basilica.deployment(
name="sqlite-api",
port=8000,
pip_packages=["fastapi", "uvicorn"],
volumes={"/data": data},
)
def serve():
from fastapi import FastAPI
import sqlite3
import uvicorn
app = FastAPI()
DB_PATH = "/data/app.db"
def get_db():
conn = sqlite3.connect(DB_PATH)
conn.execute("""
CREATE TABLE IF NOT EXISTS items (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""")
return conn
@app.post("/items")
def create_item(name: str):
conn = get_db()
cursor = conn.execute("INSERT INTO items (name) VALUES (?)", (name,))
conn.commit()
return {"id": cursor.lastrowid, "name": name}
@app.get("/items")
def list_items():
conn = get_db()
items = conn.execute("SELECT id, name, created_at FROM items").fetchall()
return [{"id": i[0], "name": i[1], "created_at": i[2]} for i in items]
@app.delete("/items/{item_id}")
def delete_item(item_id: int):
conn = get_db()
conn.execute("DELETE FROM items WHERE id = ?", (item_id,))
conn.commit()
return {"deleted": item_id}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
## GPU Examples
### PyTorch Inference
```python
import basilica
@basilica.deployment(
name="pytorch-api",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
port=8000,
gpu_count=1,
memory="8Gi",
pip_packages=["fastapi", "uvicorn"],
)
def serve():
import torch
from fastapi import FastAPI
import uvicorn
app = FastAPI()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
@app.get("/")
def info():
return {
"device": str(device),
"cuda_available": torch.cuda.is_available(),
"cuda_version": torch.version.cuda,
}
@app.post("/matmul")
def matmul(a: list, b: list):
ta = torch.tensor(a, dtype=torch.float32, device=device)
tb = torch.tensor(b, dtype=torch.float32, device=device)
result = torch.matmul(ta, tb)
return {"result": result.cpu().tolist()}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
### Hugging Face Model
```python
import basilica
cache = basilica.Volume.from_name("hf-cache", create_if_missing=True)
@basilica.deployment(
name="sentiment-api",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
port=8000,
gpu_count=1,
memory="8Gi",
pip_packages=["fastapi", "uvicorn", "transformers", "accelerate"],
volumes={"/root/.cache/huggingface": cache},
)
def serve():
from fastapi import FastAPI
from transformers import pipeline
import uvicorn
app = FastAPI()
# Load model at startup
classifier = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
device=0, # Use GPU
)
@app.post("/analyze")
def analyze(text: str):
result = classifier(text)[0]
return {
"text": text,
"label": result["label"],
"score": result["score"],
}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
### Text Generation
```python
import basilica
cache = basilica.Volume.from_name("model-cache", create_if_missing=True)
@basilica.deployment(
name="text-gen",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
port=8000,
gpu_count=1,
memory="16Gi",
pip_packages=["fastapi", "uvicorn", "transformers", "accelerate"],
volumes={"/root/.cache/huggingface": cache},
)
def serve():
from fastapi import FastAPI
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import uvicorn
app = FastAPI()
model_name = "microsoft/phi-2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
device_map="auto",
)
@app.post("/generate")
def generate(prompt: str, max_tokens: int = 100):
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=max_tokens,
do_sample=True,
temperature=0.7,
)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)
return {"prompt": prompt, "generated": text}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
## Background Jobs
### Long-Running Task
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="batch-job",
source="""
import time
from pathlib import Path
output_file = Path('/data/results.txt')
for i in range(100):
print(f"Processing batch {i+1}/100")
time.sleep(1) # Simulate work
output_file.write_text(f"Completed batch {i+1}")
print("Job complete!")
output_file.write_text("All batches complete!")
""",
storage=True,
ttl_seconds=3600, # Auto-delete after 1 hour
)
# Check logs to monitor progress
print(deployment.logs())
```
### Scheduled Worker
```python
import basilica
@basilica.deployment(
name="scheduler",
port=8000,
pip_packages=["fastapi", "uvicorn", "apscheduler"],
)
def serve():
from fastapi import FastAPI
from apscheduler.schedulers.background import BackgroundScheduler
from datetime import datetime
import uvicorn
app = FastAPI()
scheduler = BackgroundScheduler()
job_history = []
def scheduled_task():
timestamp = datetime.now().isoformat()
job_history.append(timestamp)
print(f"Job executed at {timestamp}")
scheduler.add_job(scheduled_task, 'interval', minutes=5)
scheduler.start()
@app.get("/")
def status():
return {
"running": scheduler.running,
"jobs": len(scheduler.get_jobs()),
"history": job_history[-10:], # Last 10 executions
}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
## API Patterns
### REST API with CRUD
```python
import basilica
data = basilica.Volume.from_name("api-data", create_if_missing=True)
@basilica.deployment(
name="crud-api",
port=8000,
pip_packages=["fastapi", "uvicorn", "pydantic"],
volumes={"/data": data},
)
def serve():
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import json
from pathlib import Path
import uvicorn
app = FastAPI()
DB_FILE = Path("/data/items.json")
class Item(BaseModel):
name: str
price: float
quantity: int = 0
def load_db():
if DB_FILE.exists():
return json.loads(DB_FILE.read_text())
return {}
def save_db(data):
DB_FILE.write_text(json.dumps(data, indent=2))
@app.get("/items")
def list_items():
return load_db()
@app.get("/items/{item_id}")
def get_item(item_id: str):
db = load_db()
if item_id not in db:
raise HTTPException(404, "Item not found")
return db[item_id]
@app.post("/items/{item_id}")
def create_item(item_id: str, item: Item):
db = load_db()
db[item_id] = item.dict()
save_db(db)
return {"id": item_id, **item.dict()}
@app.put("/items/{item_id}")
def update_item(item_id: str, item: Item):
db = load_db()
if item_id not in db:
raise HTTPException(404, "Item not found")
db[item_id] = item.dict()
save_db(db)
return {"id": item_id, **item.dict()}
@app.delete("/items/{item_id}")
def delete_item(item_id: str):
db = load_db()
if item_id not in db:
raise HTTPException(404, "Item not found")
del db[item_id]
save_db(db)
return {"deleted": item_id}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
print(f"API docs: {deployment.url}/docs")
```
### WebSocket Server
```python
import basilica
@basilica.deployment(
name="websocket-server",
port=8000,
pip_packages=["fastapi", "uvicorn", "websockets"],
)
def serve():
from fastapi import FastAPI, WebSocket
import uvicorn
app = FastAPI()
connections = []
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
connections.append(websocket)
try:
while True:
data = await websocket.receive_text()
# Broadcast to all connections
for conn in connections:
await conn.send_text(f"Echo: {data}")
except:
connections.remove(websocket)
@app.get("/")
def info():
return {"connections": len(connections)}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
## Next Steps
* [Deploy LLM inference with vLLM](/inference/vllm)
* [Deploy LLM inference with SGLang](/inference/sglang)
* [Learn about the deployment API](/sdk/deployments)
# GPU Configuration
# GPU Configuration
Access NVIDIA's finest silicon. A100s, H100s, B200s. No waitlists, no quotas.
## Quick Start
Request a GPU with a few parameters:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="gpu-app",
source="app.py",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
memory="8Gi",
)
```
## GPU Parameters
| Parameter | Type | Description |
| ------------------- | ----------- | ---------------------- |
| `gpu_count` | `int` | Number of GPUs (1-8) |
| `gpu_models` | `List[str]` | Acceptable GPU models |
| `min_cuda_version` | `str` | Minimum CUDA version |
| `min_gpu_memory_gb` | `int` | Minimum GPU VRAM in GB |
## Available GPUs
| Model | VRAM | CUDA Compute |
| ---------------- | ---- | ------------ |
| NVIDIA RTX A4000 | 16GB | 8.6 |
GPU availability varies by region and demand. The scheduler will find a suitable GPU matching your requirements.
## Basic GPU Deployment
### Single GPU
```python
deployment = client.deploy(
name="gpu-test",
source="""
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"Device count: {torch.cuda.device_count()}")
print(f"Device name: {torch.cuda.get_device_name(0)}")
# Keep running
import time
while True:
time.sleep(60)
""",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
memory="8Gi",
)
```
### Specific GPU Model
```python
deployment = client.deploy(
name="specific-gpu",
source="app.py",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
gpu_models=["NVIDIA-RTX-A4000"], # Only this model
memory="8Gi",
)
```
### Multiple Acceptable Models
```python
deployment = client.deploy(
name="flexible-gpu",
source="app.py",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
gpu_models=["H100", "A100", "NVIDIA-RTX-A4000"], # Any of these
memory="8Gi",
)
```
### CUDA Version Requirements
```python
deployment = client.deploy(
name="cuda12-app",
source="app.py",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
min_cuda_version="12.0", # Requires CUDA 12.0+
memory="8Gi",
)
```
### VRAM Requirements
```python
deployment = client.deploy(
name="large-model",
source="app.py",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
min_gpu_memory_gb=16, # Requires 16GB+ VRAM
memory="16Gi",
)
```
## Multi-GPU Deployments
### Multiple GPUs
```python
deployment = client.deploy(
name="multi-gpu",
source="app.py",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=4,
memory="32Gi",
)
```
### Tensor Parallelism
For large models that don't fit on a single GPU:
```python
deployment = client.deploy(
name="llm-tp4",
source="""
from vllm import LLM
# Model sharded across 4 GPUs
llm = LLM(
model="meta-llama/Llama-2-70b",
tensor_parallel_size=4,
)
# Serve...
""",
image="vllm/vllm-openai:latest",
gpu_count=4,
memory="128Gi",
)
```
## Decorator Pattern
Use the decorator for GPU deployments:
```python
import basilica
@basilica.deployment(
name="pytorch-app",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
port=8000,
gpu_count=1,
gpu_models=["NVIDIA-RTX-A4000"],
memory="8Gi",
)
def serve():
import torch
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
info = {
"cuda_available": torch.cuda.is_available(),
"device_count": torch.cuda.device_count(),
"device_name": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
"cuda_version": torch.version.cuda,
}
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps(info).encode())
HTTPServer(('', 8000), Handler).serve_forever()
deployment = serve()
```
## Common GPU Images
| Use Case | Image |
| ----------- | ----------------------------------------------- |
| PyTorch | `pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime` |
| TensorFlow | `tensorflow/tensorflow:2.14.0-gpu` |
| vLLM | `vllm/vllm-openai:latest` |
| SGLang | `lmsysorg/sglang:latest` |
| NVIDIA Base | `nvidia/cuda:12.1-runtime-ubuntu22.04` |
## GPU with Persistent Storage
Cache model weights to avoid re-downloading:
```python
import basilica
model_cache = basilica.Volume.from_name("model-cache", create_if_missing=True)
@basilica.deployment(
name="cached-model",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
memory="16Gi",
volumes={"/root/.cache/huggingface": model_cache}
)
def serve():
from transformers import AutoModelForCausalLM
# First run: downloads to cache
# Subsequent runs: loads from cache
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b",
device_map="auto",
)
# Serve model...
```
## PyTorch Example
Full PyTorch inference server:
```python
import basilica
@basilica.deployment(
name="pytorch-inference",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
port=8000,
gpu_count=1,
memory="8Gi",
pip_packages=["fastapi", "uvicorn"],
)
def serve():
import torch
from fastapi import FastAPI
import uvicorn
app = FastAPI()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
@app.get("/")
def info():
return {
"device": str(device),
"cuda": torch.cuda.is_available(),
"device_name": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
}
@app.post("/predict")
def predict(data: dict):
# Your inference logic here
tensor = torch.tensor(data["input"]).to(device)
result = tensor * 2 # Example operation
return {"output": result.cpu().tolist()}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
print(f"API: {deployment.url}")
```
## Resource Recommendations
| Model Size | GPU Count | GPU VRAM | Memory |
| -------------------- | --------- | -------- | ------ |
| Small (\< 3B params) | 1 | 8GB | 8Gi |
| Medium (3-7B params) | 1 | 16GB | 16Gi |
| Large (7-13B params) | 1-2 | 24GB+ | 32Gi |
| XL (30-70B params) | 2-4 | 40GB+ | 64Gi+ |
GPU scheduling may take longer than CPU-only deployments. Set an appropriate `timeout` value for large deployments.
## Debugging GPU Issues
### Check CUDA Availability
```python
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
print(f"cuDNN version: {torch.backends.cudnn.version()}")
```
### Check GPU Memory
```python
import torch
if torch.cuda.is_available():
print(f"Total memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
print(f"Allocated: {torch.cuda.memory_allocated(0) / 1e9:.1f} GB")
print(f"Cached: {torch.cuda.memory_reserved(0) / 1e9:.1f} GB")
```
### Common Issues
| Issue | Solution |
| ------------------ | ------------------------------------------------- |
| CUDA not available | Ensure using GPU image and `gpu_count >= 1` |
| Out of memory | Increase `min_gpu_memory_gb` or use multiple GPUs |
| Slow startup | Use persistent storage to cache models |
| Driver mismatch | Check CUDA version compatibility with image |
## Next Steps
* [Deploy vLLM inference servers](/inference/vllm)
* [Deploy SGLang inference servers](/inference/sglang)
* [Browse GPU examples](/sdk/examples#gpu-examples)
# Overview
# Overview
Deploy applications and manage infrastructure from Python.
## Installation
```bash
pip install basilica-sdk
```
Or with uv:
```bash
uv pip install basilica-sdk
```
## Quick Start
```python
import basilica
@basilica.deployment(name="hello-world", port=8000)
def serve():
from http.server import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
self.wfile.write(b'Hello World!')
HTTPServer(('', 8000), Handler).serve_forever()
deployment = serve()
print(f"Live at: {deployment.url}")
```
## Client Initialization
The `BasilicaClient` is your entry point.
```python
from basilica import BasilicaClient
# Auto-detect from BASILICA_API_TOKEN environment variable
client = BasilicaClient()
# Explicit configuration
client = BasilicaClient(
base_url="https://api.basilica.ai",
api_key="basilica_..."
)
```
**API key sources (in order of preference):**
1. `api_key` parameter passed to constructor
2. `BASILICA_API_TOKEN` environment variable
3. `BASILICA_API_URL` for custom endpoint (defaults to `https://api.basilica.ai`)
To obtain an API token, use the CLI:
```bash
basilica tokens create
```
## Core Features
### Deployment Methods
| Method | Description |
| --------------------------------------------- | ---------------------------------------- |
| [`client.deploy()`](/sdk/deployments) | Deploy any application with full control |
| [`client.deploy_vllm()`](/inference/vllm) | Deploy vLLM inference server |
| [`client.deploy_sglang()`](/inference/sglang) | Deploy SGLang inference server |
### Management Methods
| Method | Description |
| ------------------ | ------------------------ |
| `client.get(name)` | Get a deployment by name |
| `client.list()` | List all deployments |
### Deployment Object
Every deployment returns a `Deployment` object:
```python
deployment = client.deploy(name="my-app", source="app.py")
# Properties
deployment.url # Public URL
deployment.name # Deployment name
deployment.state # Current state
deployment.namespace # Kubernetes namespace
deployment.user_id # Owner user ID
deployment.created_at # Creation timestamp
# Methods
deployment.status() # Get detailed status
deployment.logs() # Get container logs
deployment.delete() # Delete the deployment
deployment.refresh() # Refresh cached state
```
## Environment Variables
```bash
# Required for authentication
export BASILICA_API_TOKEN="basilica_..."
# Optional custom endpoint
export BASILICA_API_URL="https://api.basilica.ai"
```
## Error Handling
The SDK provides detailed error information.
```python
from basilica import (
BasilicaError,
DeploymentTimeout,
DeploymentFailed,
DeploymentNotFound,
ValidationError,
)
try:
deployment = client.deploy("my-app", source="app.py")
except DeploymentTimeout as e:
print(f"Deployment timed out after {e.timeout_seconds}s")
print(f"Last state: {e.last_state}")
except DeploymentFailed as e:
print(f"Deployment failed: {e.reason}")
except DeploymentNotFound as e:
print(f"Deployment {e.instance_name} not found")
except ValidationError as e:
print(f"Invalid parameter {e.field}: {e.value}")
except BasilicaError as e:
print(f"Error: {e.message}")
print(f"Retryable: {e.retryable}")
```
**Exception hierarchy:**
```
BasilicaError (base)
├── AuthenticationError
├── AuthorizationError
├── ValidationError
├── DeploymentError
│ ├── DeploymentNotFound
│ ├── DeploymentTimeout
│ └── DeploymentFailed
├── ResourceError
├── StorageError
├── NetworkError
├── RateLimitError
└── SourceError
```
## Next Steps
* **[Deployments](/sdk/deployments):** Deploy applications
* **[Storage](/sdk/storage):** Persist data across restarts
* **[GPU Workloads](/sdk/gpu):** Deploy on GPU hardware
* **[Examples](/sdk/examples):** More patterns and use cases
# Storage
# Storage
Persistent storage that survives container restarts. Your data lives on, backed by Cloudflare R2.
## Quick Start
Enable storage with a single parameter:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="app",
source="app.py",
storage=True, # Mounts at /data
)
```
Then use `/data` in your application:
```python
from pathlib import Path
# Write data
Path('/data/counter.txt').write_text('42')
# Read data
count = Path('/data/counter.txt').read_text()
```
## Storage Options
### Default Mount Path
```python
deployment = client.deploy(
name="app",
source="app.py",
storage=True, # Mounts at /data (default)
)
```
### Custom Mount Path
```python
deployment = client.deploy(
name="app",
source="app.py",
storage="/cache", # Mounts at /cache
)
```
### Named Volumes (Decorator Pattern)
For more control, use named volumes with the decorator:
```python
import basilica
# Create or get a named volume
cache = basilica.Volume.from_name("model-cache", create_if_missing=True)
@basilica.deployment(
name="app",
volumes={"/models": cache}
)
def serve():
from pathlib import Path
models_dir = Path("/models")
# ...
```
## Volume API
### Creating Volumes
```python
import basilica
# Get existing volume or create if missing
volume = basilica.Volume.from_name(
name="my-volume", # Volume name (bucket identifier)
create_if_missing=True # Create if doesn't exist
)
```
### Multiple Volumes
Mount multiple volumes to different paths:
```python
import basilica
data = basilica.Volume.from_name("app-data", create_if_missing=True)
cache = basilica.Volume.from_name("model-cache", create_if_missing=True)
@basilica.deployment(
name="ml-app",
volumes={
"/data": data,
"/cache": cache,
}
)
def serve():
# /data for user data
# /cache for model weights
pass
```
## Common Patterns
### Persistent Counter
```python
from basilica import BasilicaClient
from pathlib import Path
client = BasilicaClient()
deployment = client.deploy(
name="counter",
source="""
from pathlib import Path
from http.server import HTTPServer, BaseHTTPRequestHandler
COUNTER_FILE = Path('/data/count.txt')
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
# Read current count
count = int(COUNTER_FILE.read_text()) if COUNTER_FILE.exists() else 0
count += 1
# Save new count
COUNTER_FILE.write_text(str(count))
self.send_response(200)
self.end_headers()
self.wfile.write(f'Count: {count}'.encode())
HTTPServer(('', 8000), Handler).serve_forever()
""",
port=8000,
storage=True,
)
```
### Model Caching
Cache downloaded models to avoid re-downloading:
```python
import basilica
cache = basilica.Volume.from_name("huggingface-cache", create_if_missing=True)
@basilica.deployment(
name="llm",
image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
gpu_count=1,
memory="16Gi",
volumes={"/root/.cache/huggingface": cache}
)
def serve():
from transformers import AutoModelForCausalLM, AutoTokenizer
# Model downloads once, then cached in volume
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")
# Serve model...
```
### SQLite Database
Use SQLite for lightweight persistence:
```python
from basilica import BasilicaClient
client = BasilicaClient()
deployment = client.deploy(
name="app",
source="""
import sqlite3
from pathlib import Path
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
DB_PATH = '/data/app.db'
def init_db():
conn = sqlite3.connect(DB_PATH)
conn.execute('CREATE TABLE IF NOT EXISTS visits (id INTEGER PRIMARY KEY, ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP)')
conn.commit()
return conn
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
conn = init_db()
conn.execute('INSERT INTO visits DEFAULT VALUES')
conn.commit()
count = conn.execute('SELECT COUNT(*) FROM visits').fetchone()[0]
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({'visits': count}).encode())
HTTPServer(('', 8000), Handler).serve_forever()
""",
port=8000,
storage=True,
)
```
### File Upload Service
```python
import basilica
uploads = basilica.Volume.from_name("user-uploads", create_if_missing=True)
@basilica.deployment(
name="upload-service",
port=8000,
pip_packages=["fastapi", "uvicorn", "python-multipart"],
volumes={"/uploads": uploads}
)
def serve():
from fastapi import FastAPI, UploadFile
from pathlib import Path
import uvicorn
app = FastAPI()
UPLOAD_DIR = Path("/uploads")
@app.post("/upload")
async def upload(file: UploadFile):
path = UPLOAD_DIR / file.filename
with open(path, "wb") as f:
f.write(await file.read())
return {"path": str(path)}
@app.get("/files")
def list_files():
return {"files": [f.name for f in UPLOAD_DIR.iterdir()]}
uvicorn.run(app, host="0.0.0.0", port=8000)
deployment = serve()
```
## Technical Details
### Storage Backend
| Property | Value |
| ------------- | --------------------- |
| Backend | Cloudflare R2 |
| Mount Type | FUSE filesystem |
| Sync Interval | 1000ms |
| Cache Size | 1024MB (configurable) |
### Readiness Detection
The storage system creates a marker file when ready:
```python
from pathlib import Path
# Wait for storage to be ready
MARKER = Path('/data/.fuse_ready')
while not MARKER.exists():
time.sleep(0.1)
# Storage is now ready
```
The SDK handles readiness automatically. You only need manual checks for custom images or scripts.
### Performance Considerations
* **Write latency**: Writes are cached locally and synced to R2 asynchronously
* **Read latency**: First read fetches from R2, subsequent reads are cached
* **Sync interval**: Data syncs every 1 second by default
* **Large files**: Consider streaming for files larger than cache size
### Volume Naming
Volume names must be:
* Lowercase letters, numbers, and hyphens only
* 3-63 characters long
* Unique within your account
```python
# Good names
basilica.Volume.from_name("my-data", create_if_missing=True)
basilica.Volume.from_name("cache-v2", create_if_missing=True)
basilica.Volume.from_name("prod-models", create_if_missing=True)
# Invalid names (will fail)
basilica.Volume.from_name("My Data") # No spaces, uppercase
basilica.Volume.from_name("cache_v2") # No underscores
basilica.Volume.from_name("a") # Too short
```
## Next Steps
* [Deploy GPU workloads](/sdk/gpu) with persistent model storage
* [Browse storage examples](/sdk/examples#storage-examples)