An AI Glossary for Engineers

August 10, 2026 Khimananda Oli 8 min read Virtualization

By Khimananda Oli | Last reviewed: August 2026

Marketing teams define AI by capabilities; engineers must define it by constraints, costs, and failure modes. This AI glossary for engineers strips away the hype to focus on the operational realities of deploying machine learning systems in production. Whether you are integrating an API or managing GPU clusters, understanding these terms prevents costly architectural mistakes. For a deeper look at how these concepts apply to operations specifically, see my guide on AIOps and modern infrastructure automation.

Quick answer: An AI glossary for engineers is a technical reference defining machine learning terms through the lens of infrastructure, latency, cost, and reliability rather than marketing. It distinguishes between probabilistic generation and deterministic execution, focusing on implementation trade-offs like context windows, retrieval accuracy, and inference compute requirements.

Core AI engineering stack: data flows from raw input through embeddings and vector storage to LLM inference and final application output.

What are the core infrastructure terms in an AI glossary for engineers?

When building production systems, vague definitions lead to capacity planning failures. You need precise mental models for the components that consume budget and dictate latency. These foundational terms appear in every architecture review and incident postmortem involving intelligent systems.

Tokens and Context Windows

A token is the atomic unit of processing for a Large Language Model (LLM), not a word. In English, one token averages 0.75 words, but for code or Nepali text, this ratio shifts significantly due to byte-pair encoding efficiency. Pricing and rate limits are universally denominated in tokens. The context window is the maximum number of tokens a model can attend to in a single request, including both input prompt and generated output. Exceeding this limit causes truncation or errors. While some 2026 models advertise million-token contexts, effective recall often degrades beyond 128k tokens without specific retrieval augmentation strategies.

Inference vs. Training

Training is the computationally expensive process of adjusting model weights on massive datasets to learn patterns; it happens offline, often taking weeks on H100/H200 clusters. Inference is the production phase where the frozen model processes user requests to generate predictions or text. As an engineer, you spend 99% of your time optimizing inference: managing KV caches, quantizing weights to INT8/FP8, and balancing throughput against latency. Confusing these two phases leads to disastrous resource allocation, such as provisioning training-grade GPUs for simple API serving.

Embeddings and Vector Databases

Embeddings are dense numerical representations of semantic meaning, typically floating-point arrays of 768 to 3072 dimensions. They transform unstructured text into coordinates where similar concepts cluster together. A vector database (like pgvector, Pinecone, or Weaviate) stores these embeddings and performs Approximate Nearest Neighbor (ANN) searches. Unlike traditional SQL databases optimized for exact matches, vector DBs optimize for similarity search speed using indexes like HNSW. Understanding the trade-off between index build time, memory usage, and recall accuracy is critical for RAG performance.

How do RAG, fine-tuning, and prompt engineering differ?

Choosing the right adaptation strategy is the most common architectural decision point. Each approach has distinct cost profiles, maintenance burdens, and failure modes. Use this comparison to justify your technical choices to stakeholders.

Strategy	Best For	Latency Impact	Cost Profile	Knowledge Freshness
RAG	Factual accuracy, proprietary data, auditability	+50–200ms (retrieval step)	Low (inference only + vector storage)	Real-time (update source docs)
Fine-Tuning	Style transfer, complex formats, domain jargon	None (native generation)	High upfront (training run) + hosting	Stale until retrained
Prompt Engineering	Rapid prototyping, logic routing, guardrails	Negligible	Lowest (just token costs)	Immediate
Long Context	Document analysis, codebase reasoning	Linear scaling with size	Medium-High (quadratic attention cost)	Per-request injection

Retrieval-Augmented Generation (RAG) grounds model outputs in external data. Instead of relying solely on parametric memory, the system retrieves relevant chunks from a vector store and injects them into the prompt. This reduces hallucinations and allows citation. However, RAG introduces complexity: chunk size, overlap, and reranking algorithms directly impact answer quality. If your retrieval is poor, the best model in the world will give wrong answers confidently. For teams implementing this pattern, I detail specific database trade-offs in vector databases for RAG: pgvector vs Pinecone.

Fine-tuning modifies model weights via supervised learning on a curated dataset. Contrary to popular belief, fine-tuning does not reliably teach new facts; it teaches behavior, tone, and schema adherence. Use it when RAG consistently fails to produce the correct output format or when you need to distill a larger model's capabilities into a smaller, cheaper one. The operational overhead includes maintaining training pipelines, evaluation datasets, and versioned model artifacts.

Architectural decision flow: routing queries between RAG for factual grounding and fine-tuned models for specialized formatting or style.

What defines AI agents and agentic workflows in production?

The term "agent" is frequently overloaded. In engineering practice, an AI agent is a system where an LLM operates within a feedback loop, possessing three specific capabilities: tool use, memory, and autonomous planning. A chatbot that answers questions is not an agent. A system that reads a Jira ticket, queries Datadog, analyzes logs, and drafts a postmortem is an agent.

Tool Use and Function Calling

This is the mechanism allowing models to interact with external systems. The model outputs structured JSON matching a predefined schema instead of natural language. Your application layer parses this JSON, executes the corresponding API call or script, and feeds the result back into the context. Reliability here depends entirely on schema clarity and error handling. If your function definitions are ambiguous, the model will hallucinate parameters. Always validate agent outputs against strict schemas before execution, especially in automated DevOps tasks where incorrect commands can cause outages.

Memory Systems

Agents require memory beyond the ephemeral context window. Short-term memory is the current conversation buffer. Long-term memory involves persisting interactions, user preferences, or learned facts to a database for retrieval in future sessions. Working memory refers to scratchpads or intermediate state storage used during multi-step reasoning. Implementing robust memory management is what separates demo-ware from production systems that maintain coherence over weeks or months.

Orchestration Patterns

Single-agent loops often fail at complex tasks. Production systems use orchestration patterns like ReAct (Reason + Act), Plan-and-Solve, or multi-agent hierarchies. In a hierarchical setup, a "manager" agent decomposes goals while specialized "worker" agents execute subtasks. This modularity improves debuggability and allows mixing different models (e.g., a cheap router with expensive specialists). Guardrails and human-in-the-loop approvals are mandatory for any agent with write access to production environments.

How do MLOps and LLMOps differ from traditional DevOps?

While sharing CI/CD DNA, AI operations introduce non-determinism that breaks standard testing assumptions. Code is deterministic; model outputs are probabilistic. This fundamental shift requires new vocabulary and practices.

Evaluation (Evals): Replaces unit tests. Instead of asserting exact equality, you assert semantic similarity, rubric compliance, or safety thresholds. Evals must run automatically in CI against golden datasets. Without automated evals, you cannot safely deploy model updates.
Guardrails: Input/output filters that enforce policy boundaries. These include PII detection, toxicity filtering, and schema validation. Guardrails act as the firewall between the stochastic model and your deterministic business logic.
Observability for LLMs: Traditional metrics (CPU, RAM) are insufficient. You need tracing for token usage, latency per step, retrieval relevance scores, and semantic drift detection. Tools like LangSmith, Arize, or OpenTelemetry extensions provide visibility into the "black box" of generation.
Model Registry & Versioning: Models are large binary artifacts, not just code pointers. Tracking lineage between training data, hyperparameters, and deployed weights is essential for reproducibility and rollback. Treat model versions with the same rigor as container images.

LLMOps feedback loop: production telemetry drives continuous evaluation and guardrail refinement in the CI pipeline.

Why does terminology precision matter for AI system reliability?

Ambiguity in AI discussions causes incidents. When a stakeholder asks for "smarter responses," they might mean better retrieval (RAG), different tone (fine-tuning), or fewer refusals (guardrail tuning). Each solution has radically different timelines and risks. Using precise terms from this AI glossary for engineers aligns expectations and prevents scope creep. It also aids in vendor evaluation: knowing the difference between "context length" and "effective context" stops you from buying capacity you cannot actually use.

Furthermore, security and compliance depend on accurate definitions. SOC 2 and ISO 27001 audits require you to define data boundaries. If you cannot articulate whether customer data enters the model's training set or stays within the inference context, you cannot pass an audit. Precise terminology enables accurate threat modeling and data residency documentation. In Nepal's growing tech sector, where many companies serve global clients, demonstrating this level of technical maturity is a competitive differentiator.

Next Steps for Engineering Teams

Mastering this vocabulary is the first step toward building reliable AI systems. Audit your current architecture diagrams: replace generic "AI" boxes with specific components like "Embedding Service," "Vector Index," or "Guardrail Layer." Establish shared definitions with your product team to streamline requirements gathering. Start tracking the metrics that actually matter—eval pass rates, retrieval latency, and token costs per transaction—rather than vanity metrics. If your team needs help translating these concepts into a production-ready infrastructure strategy or preparing for an AI-focused compliance audit, reach out to discuss your specific implementation challenges.

Frequently Asked Questions

RAG retrieves external context at inference time without modifying weights, while fine-tuning updates model parameters using training data. Choose RAG for dynamic knowledge and fine-tuning for behavioral adaptation or domain-specific syntax patterns in 2026 production systems.

Multiply input and output tokens by provider-specific rates listed in pricing tables. Use tiktoken or provider SDKs to count tokens before sending requests. Budget for context window expansion during multi-turn conversations or large document processing tasks.

Temperature scales logits before sampling to control randomness. Lower values produce deterministic outputs suitable for code generation, while higher values increase creativity. Most engineering tasks use 0.0 to 0.3 for reproducible results in CI/CD pipelines.

Shared terminology prevents miscommunication between ML engineers and infrastructure teams during deployment. Standardized definitions accelerate onboarding, reduce ticket back-and-forth, and ensure consistent configuration across staging and production environments when integrating new model services.

Quantization reduces model precision from FP16 to INT8 or INT4 to decrease memory footprint and latency. Apply it when deploying on edge devices or fitting larger models into limited GPU VRAM. Expect minor accuracy trade-offs requiring validation testing.

Embeddings map text to dense vector representations capturing semantic meaning rather than exact string matches. This enables similarity search across paraphrased content. Engineers use vector databases like pgvector or Qdrant to store and query these high-dimensional representations efficiently.

Guardrails are validation layers that filter inputs and outputs for safety, format compliance, or policy adherence. Implement them as middleware using tools like Guardrails AI or NeMo to prevent prompt injection, PII leakage, or hallucinated responses in production APIs.

Use RLHF when aligning model outputs with human preferences that lack explicit ground truth labels. Supervised fine-tuning suffices for structured tasks with clear correct answers. RLHF adds complexity through reward modeling and PPO training loops requiring specialized infrastructure.

Store model metadata, hyperparameters, and dataset hashes in Git while keeping weights in object storage or model registries like MLflow. Tag releases semantically and track lineage between training runs, evaluation benchmarks, and deployed endpoints for auditability and rollback capability.

Context window overflow occurs when combined prompt, history, and response tokens exceed model limits. Truncate conversation history, summarize prior exchanges, or switch to models with larger windows. Monitor token counts programmatically to handle dynamic content gracefully without runtime failures.

Yes, if you conduct security audits, apply guardrails, and comply with license terms. Open-weight models offer transparency and self-hosting options but require your team to manage patching, vulnerability scanning, and access controls that managed providers typically handle automatically.

Speculative decoding uses a smaller draft model to propose multiple tokens that the larger target model verifies in parallel. This reduces sequential generation steps without quality loss. Enable it in vLLM or TensorRT-LLM for 2x throughput gains on compatible hardware.

System prompts define persistent behavioral instructions separate from user messages. They establish role, tone, constraints, and output formatting rules. Keep them concise and test thoroughly since changes affect all downstream interactions without visible user-facing modifications.

Measure retrieval precision, answer faithfulness, and relevance using frameworks like RAGAS or Aries. Build golden test sets with known questions and expected citations. Track metrics over time to detect degradation after index updates or embedding model changes.

Agents autonomously plan, execute tool calls, and iterate based on observations rather than generating single responses. They maintain state across steps and handle error recovery. Implement using ReAct patterns or orchestration frameworks with strict permission boundaries and timeout safeguards.