Apache Airflow Wait Between Tasks
Apache Airflow allows you to create custom workflows that can be triggered on a recurring basis at specific time intervals based on your needs.
Some concepts to learn are
DAGs - this defines the entire workflow
Tasks - are the Individual tasks that constitute the workflow
Operators - This is an airflow-specific concept, which needs to be understood, Based on what type of tasks you would like to perform there are multiple operators. To run a bash script there is a BashOperator, to run a python code you need to have a Python Operator.
For more details, refer https://airflow.apache.org/docs/apache-airflow/stable/concepts/index.html
Above 3 are sufficient to get started, ofcourse there are others like interprocess communication, variables, process dependency management, etc.
Let's Define a Simple Workflow we want to achieve
DAG = Tasks1 >> Wait for a Day >> Task2
Above is the DAG, where two tasks are executed, second task depends on first task and is executed after a delay of one day.
Lets explore how can we achieve this,
We will create two operatators for task1 and task2 and a task for delay, that delay task will sleep for the designated period and then executed the task2.
Like so
import time
from airflow.operators.python import PythonOperator
delay_def: PythonOperator = PythonOperator
(
task_id="delay_python_task",
dag=my_dag,
python_callable=lambda: time.sleep(300))
Comments
Post a Comment