Hugging Face: Hub, routed/hosted inference, and transformers

Hugging Face is three surfaces, and you should always know which one you are on:

1. The Hub — versioned git repos for models, datasets, and Spaces. You search it, you hf download / hf upload, you read and write model cards.
2. Inference — three ways to actually run a model: the Inference Providers router (serverless, you own nothing), a dedicated Inference Endpoint (you own a deployment that autoscales), or local transformers (you own the machine).
3. The catalog — 1M+ open models you choose from by task, license, and size.

The whole skill is choosing the right surface for the job and proving it works: a 200 router response, a live endpoint URL, a pushed repo commit. If the model is open and the workflow lives on huggingface.co, you are in the right place. Operating the GPU box yourself is ../ollama/SKILL.md (your machine) or ../runpod/SKILL.md (a rented box); training weights is ../finetuning/SKILL.md.

Decision: how should I run this model?

Pick the row before you write a line of code. The cheapest mistake is standing up infra you did not need.

Situation	Use	Why
Try a model now, low/dev volume, own no infra	Inference Providers router (InferenceClient)	Fastest path; monthly credits cover dev.
CPU task: embeddings, text-ranking, text-classification, small BERT/GPT-2	provider="hf-inference"	That is exactly its remaining niche as of July 2025.
Big LLM (8B, 70B, 405B) through HF	router with a partner provider (Together/Fireworks/Cerebras/DeepInfra…)	hf-inference does not serve big LLMs — it will 404 or stall.
Steady prod traffic, need fixed latency/SLA	dedicated Inference Endpoint + scale-to-zero	Predictable, autoscaling, billed per minute.
Interactive demo or shareable GPU app	Space (Gradio + ZeroGPU)	Free-ish, public URL, GPU only while a call runs.
One-off GPU job (eval, batch convert)	hf jobs run	No standing infra; PRO feature.
Offline, data-private, or already on a GPU box	local transformers pipeline()	No network, no per-call cost.

Auth & install

pip install "huggingface_hub[inference]"   # 1.17.0; needs Python >=3.10
pip install transformers                    # 5.x line, PyTorch-first, optional/local
hf auth login                               # stores a token; or export HF_TOKEN=...

• The CLI is hf now, shaped hf <resource> <action> (hf auth login, hf download, hf upload, hf repo create, hf jobs run). huggingface-cli still runs but prints a deprecation warning — do not write it into new scripts.
• Never hardcode a hf_... token in code — tokens leak the moment the file hits git. Read from the environment instead:

import os
from huggingface_hub import InferenceClient
client = InferenceClient(api_key=os.environ["HF_TOKEN"])   # never api_key="hf_xxx"

• Token scopes: read to pull public/gated repos and run inference, write to push, fine-grained to scope to specific repos/orgs — why: a leaked read token cannot overwrite your models.

Inference Providers — the default path

One router reaches 200+ models across partner providers plus hf-inference; HF passes provider cost through with no markup. Two equivalent entry points:

# Native client — task methods, NOT the removed .post()
from huggingface_hub import InferenceClient
client = InferenceClient(api_key=os.environ["HF_TOKEN"])
out = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "One sentence on diffusion models."}],
    provider="together",          # name a partner; or omit for auto-routing
)
print(out.choices[0].message.content)

# OpenAI-compatible — same router, drop-in for existing OpenAI code
from openai import OpenAI
client = OpenAI(
    base_url="https://router.huggingface.co/v1",   # this exact host, nothing else
    api_key=os.environ["HF_TOKEN"],
)

• InferenceClient.post() was removed (dropped in hub v0.31.0). Use the task methods: chat.completions.create(), text_generation(), feature_extraction() (embeddings), text_to_image(), automatic_speech_recognition().
• Credits are real and small: Free $0.10/mo, PRO $2.00/mo, Team/Enterprise $2.00 per seat (shared). Past that you are pay-as-you-go and must buy credits. Budget accordingly — why: a chat loop on a 70B model burns the free tier in minutes.
• A Custom Provider Key bypasses HF billing entirely (the provider bills you; HF credits do not apply). For org billing, pass bill_to="org-name" (header X-HF-Bill-To).
• Full recipes (embeddings, image, ASR, streaming, rate-limit handling, the provider list) live in references/inference-providers.md.

Hub ops

hf download meta-llama/Llama-3.1-8B-Instruct --include "*.safetensors"
hf repo create my-org/my-model --repo-type model
hf upload my-org/my-model ./out --commit-message "v1 weights"

from huggingface_hub import snapshot_download
path = snapshot_download("BAAI/bge-small-en-v1.5")   # full repo, cached, resumable

• Gated models (Llama, Gemma, many others) need you to accept terms on the model page first, then a token with read access — otherwise the download 403s.
• A model card is a README.md with YAML front-matter (license, pipeline_tag, tags, base_model). Ship one on every upload — why: an uncarded repo is unsearchable and unusable by anyone but you. Command map and hf jobs run details in references/hub-and-cli.md.

Choosing a model

Filter the Hub by task + license + size + recent downloads, then read the card before you commit. Match the model to your constraint; do not grab whatever is trending.

• Check the license: Apache-2.0/MIT are permissive; Llama/Gemma carry commercial terms and are gated; "non-commercial"/"research-only" cards mean you cannot ship them.
• Check size vs target: a 70B will not fit a single A10G; an embedding model belongs on CPU.
• Check context length and intended use in the card — the headline number is not always the usable one.

Dedicated Inference Endpoints — when to graduate

Move off the router when you need fixed latency/SLA, or the router's PAYG cost stops being predictable. An Endpoint is your own autoscaling deployment.

• Pricing: CPU from ~$0.032/core/hr, GPU from ~$0.50/hr (A10G ~$1.00/hr, H100 ~$6.40–8.00/hr), billed per minute even though shown hourly.
• Enable scale-to-zero for bursty traffic — it parks at $0 when idle and cold-starts on the next request. A bursty 100–1000 req/day workload typically lands at $20–60/mo.
• Deploy from the UI or with huggingface_hub (create_inference_endpoint(...)). Config and a cost worksheet are in references/endpoints-and-spaces.md.

Spaces + ZeroGPU

A Space hosts a demo app with a public URL. ZeroGPU grabs an H200 MIG slice (~70GB) only while a decorated function runs, then releases it.

import spaces
@spaces.GPU                       # GPU acquired for this call only
def generate(prompt: str) -> str:
    ...

• ZeroGPU is Gradio-SDK only — Streamlit/Docker/static Spaces cannot use it. PRO ($9/mo) gives 8x daily quota, queue priority, and up to 10 owned ZeroGPU Spaces. Details in references/endpoints-and-spaces.md.

Local transformers

from transformers import pipeline
pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct",
                device_map="auto", torch_dtype="auto")
print(pipe("Hello", max_new_tokens=64)[0]["generated_text"])

• pipeline("task", model=...) for quick use; AutoModelForCausalLM.from_pretrained(...) when you need control over generation/quantization. Set device_map/torch_dtype explicitly.
• Use local only when you are offline, data-private, or already on a GPU. Otherwise the router is far less ops than babysitting CUDA and weights.

Anti-patterns

Anti-pattern	Why it bites	Do instead
InferenceClient.post(...)	Removed in hub v0.31.0; raises	Task methods: chat.completions.create(), feature_extraction()
provider="hf-inference" for a 70B/405B LLM	CPU niche; 404s or stalls	Route to a partner provider (Together/Fireworks/Cerebras)
api_key="hf_abc123..." in code	Token leaks in git history	Read os.environ["HF_TOKEN"]
Spin up a dedicated Endpoint just to try a model	Burns money idle	Use the router first; graduate only on real traffic
Assuming router calls are free/unlimited	Free tier is $0.10/mo	Budget credits; expect PAYG
ZeroGPU under Streamlit/Docker SDK	Unsupported, silently no GPU	Use the Gradio SDK
huggingface-cli ... in new scripts	Deprecated, warns	Use hf ...
OpenAI base URL other than https://router.huggingface.co/v1	Won't reach the HF router	Use that exact host

verify.sh

scripts/verify.sh [TARGET] is a static, read-only linter (no network, no token). It flags the hard violations above — .post(, hardcoded hf_ tokens, big-LLM-to-hf-inference, wrong router host — and warns on legacy huggingface-cli. It exits 0 on a clean or empty target.