DataTerminology.comNEW
Answering questions using only the model’s internal parameters, without consulting external documents or retrieval.
DESCRIPTION
Closed-book QA evaluates how much factual knowledge is stored within model parameters. Systems like T5 and GPT are tested to answer without retrieval. It highlights memorization limits, hallucination risks, and the value of fine-tuning. Comparisons with open-book and RAG approaches reveal trade-offs in accuracy and latency. Robust evaluation requires curated datasets and careful prompt design.
ALTERNATIVE DEFINITIONS
arXiv
We consider closed-book question answering, where the model must answer factoid questions without access to any external knowledge sources.
Defines the task as answering without external sources, emphasizing parametric knowledge.
Hugging Face
In closed-book QA, a model answers questions without retrieving documents; it relies solely on knowledge stored in its parameters.
Clarifies reliance on parametric knowledge and absence of retrieval.
RELATED TERMS
Language Model Parametric Knowledge Few-Shot Learning Prompting Knowledge Distillation Benchmarking Hallucination Fine-Tuning
COMPETING TERMS
Open-book Question Answering Retrieval-Augmented Generation
RELATIONS
- Broader: Question Answering
KNOWLEDGE AREAS
AI Data Science MLOps Testing & Validation
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer AI Product Manager Chief AI Officer
EXTERNAL SOURCES
- Wikipedia - Question answering
- Papers with Code - QA tasks and benchmarks
- TensorFlow - Transformer tutorial underpinning QA
TOP QUESTIONS
Why use Closed-book Question Answering?
It measures parametric knowledge and model recall without retrieval. Useful for low-latency scenarios and capability benchmarking.
Category: Relevancy
What are limitations of Closed-book Question Answering?
It can hallucinate, lacks up-to-date knowledge, and struggles with long-tail facts compared to retrieval-based methods.
Category: Risks
How do you improve Closed-book Question Answering performance?
Fine-tune on QA corpora, use instruction tuning, and calibrate outputs; consider model size and training data coverage.
Category: HowTo
How does Closed-book Question Answering compare to RAG?
RAG improves factuality and freshness via retrieval, at the cost of complexity and latency; closed-book is simpler and faster.
Category: Alternatives
What datasets evaluate Closed-book Question Answering?
Natural Questions, TriviaQA, and WebQuestions are common benchmarks; curated subsets remove retrieval to test parametric knowledge.
Category: Testing
What is a common mistake with Closed-book Question Answering?
Assuming high accuracy without retrieval; rigorous evaluation often reveals knowledge gaps and hallucinations.
Category: Mistakes
How should outputs be validated in Closed-book Question Answering?
Use confidence scores, self-consistency checks, and post-hoc verification to mitigate hallucinations.
Category: BestPractices
Is Closed-book Question Answering suitable for compliance-sensitive domains?
Generally risky without retrieval and citations; add safeguards or switch to RAG for verifiability.
Category: Compliance
What is a striking analogy for Closed-book Question Answering?
It’s like taking an exam without notes—answers rely purely on what you’ve memorized.
Category: Example