🛰️Data Source

NEW

Also known as: Source, Data Origin

The origin or location from which data is obtained for ingestion, processing, or analysis.

DESCRIPTION

A data source can be any system, service, or artifact providing data: databases, files, APIs, sensors, or streams. Integration tools model sources with connection details, schemas, and access policies. Understanding source semantics, data contracts, and update patterns is essential for reliable pipelines. Sources can be batch or streaming and may require incremental extraction. Source governance covers consent, security, and change management.

ALTERNATIVE DEFINITIONS

Wikipedia

In computing, a data source is the location from which data used by a program or system originates.

Establishes a general computing-centered definition of origin.

Source

Microsoft Learn

In Azure Data Factory, a dataset represents the data structure within the data store that is used as inputs or outputs of a pipeline activity.

Clarifies how tooling models sources and their structures.

Source

RELATED TERMS

Source Data Store Middleware API Sensor Database Streaming Source Webhooks Dataset Queue File

COMPETING TERMS

Sink Data Store

KNOWLEDGE AREAS

Data Integration & Interoperability Data Engineering Data Management

TARGET AUDIENCE

Data Engineer Analytics Engineer Data Architect Information Analyst

EXTERNAL SOURCES

• Wikipedia - Data source
• Microsoft - Modeling sources and datasets
• Airbyte - Catalog of data source connectors

TOP QUESTIONS

How do you identify a Data Source?

Map upstream systems that produce required data—databases, APIs, queues, or files—and document ownership, schemas, and SLAs.

Category: Getting Started

What’s the difference between a Data Source and a Source Data Store?

A data source is the origin conceptually; a source data store emphasizes the concrete storage system providing the data.

Category: Differences

How should you secure Data Sources?

Use least-privilege access, secret management, network segmentation, and audit trails for access and extraction.

Category: Best Practices

How to handle unreliable Data Sources?

Implement retries with backoff, dead-letter queues, monitoring, and reconciliation against source-of-truth counts.

Category: Management

What metadata is essential for Data Sources?

Ownership, schema versions, change history, SLAs, data classification, and lineage enrollment.

Category: Requirements

How to manage Data Source changes?

Adopt data contracts, versioned schemas, deprecation timelines, and automated impact analysis.

Category: Management

What are common mistakes with Data Sources?

Hardcoding credentials, ignoring rate limits, and neglecting schema validation at ingestion.

Category: Mistakes

How do streaming Data Sources differ from batch sources?

Streaming provides continuous event flows with low latency; batch provides periodic snapshots or files.

Category: Differences

How to test connectivity to a Data Source?

Use health checks, connection tests, and small sample extracts with schema validation before production loads.

Category: Validation