DataTerminology.comNEW
Also known as: Scheduled Refresh, Time-based Refresh
A scheduled process that refreshes data at regular intervals, such as hourly or daily.
DESCRIPTION
Periodic data refreshment automates dataset or cache updates on a fixed schedule. It underpins batch ETL/ELT and BI dashboards with predictable freshness SLAs. Schedulers like cron or orchestration tools coordinate dependencies, retries, and notifications. Incremental strategies reduce load by processing only changes. Trade-offs include staleness between runs and resource spikes at schedule boundaries. Many platforms combine periodic refresh with event-driven or ad-hoc triggers for hybrid freshness.
ALTERNATIVE DEFINITIONS
Microsoft Power BI
With scheduled refresh, Power BI will connect to your data sources, query for updates, and load the updated data into the dataset on a configurable schedule.
Defines automated, time-based dataset refreshing in a mainstream BI tool.
Tableau
Use schedules to automate extract refreshes so that data is updated at regular intervals without manual intervention.
Emphasizes automation to avoid manual refresh.
Looker Studio Help
You can schedule your data source to refresh automatically at a specific time and frequency.
Specifies time and frequency controls for scheduled refresh.
RELATED TERMS
Batch Processing Incremental Refresh Orchestration Apache Airflow SLA Pipelines Materialized View Service Level Objective Data Freshness Cron
COMPETING TERMS
Ad-hoc Data Refreshment Streaming Data
KNOWLEDGE AREAS
Data Engineering Data Warehousing & BI Platform & Tooling Data Management
TARGET AUDIENCE
Data Engineer Analytics Engineer BI Specialist Data Architect Data Steward
EXTERNAL SOURCES
- Wikipedia - Batch processing
- Apache Airflow - Orchestration for scheduled pipelines
- Prefect - Workflow scheduling and orchestration
TOP QUESTIONS
Why use Periodic Data Refreshment?
It provides predictable freshness, repeatability, and lower operational complexity than continuous streaming for many business dashboards.
Category: Relevancy
How do I choose a schedule for Periodic Data Refreshment?
Align with business SLAs and source update frequency; avoid peak hours to reduce contention and costs.
Category: BestPractices
What are common mistakes with Periodic Data Refreshment?
Running full reloads unnecessarily, ignoring dependency ordering, and lacking retries or alerts cause failures and downtime.
Category: Mistakes
How does Periodic Data Refreshment compare to streaming?
Scheduled refresh trades latency for simplicity and cost, while streaming reduces latency with higher operational complexity.
Category: Comparison
What is incremental Periodic Data Refreshment?
It processes only changed data since the last run, reducing compute costs and window times.
Category: Practice
What tools help orchestrate Periodic Data Refreshment?
Use Airflow, Prefect, Dagster, or cloud schedulers to manage dependencies, retries, and notifications.
Category: Tools
How to monitor Periodic Data Refreshment?
Track run success, duration, data volumes, freshness, and SLA adherence; alert on anomalies.
Category: Observability
What are risks of Periodic Data Refreshment?
Data can be stale between runs; synchronized job spikes may strain infrastructure if not staggered.
Category: Risk
What is an alternative to Periodic Data Refreshment?
Ad-hoc Data Refreshment for urgent needs, or streaming ingestion for near-zero latency.
Category: Alternatives