Posts

Showing posts from July, 2022

Apache Airflow Wait Between Tasks

 Apache Airflow allows you to create custom workflows that can be triggered on a recurring basis at specific time intervals based on your needs. Some concepts to learn are DAGs - this defines the entire workflow Tasks - are the Individual tasks that constitute the workflow Operators - This is an airflow-specific concept, which needs to be understood, Based on what type of tasks you would like to perform there are multiple operators. To run a bash script there is a BashOperator, to run a python code you need to have a Python Operator. For more details, refer  https://airflow.apache.org/docs/apache-airflow/stable/concepts/index.html Above 3 are sufficient to get started, ofcourse there are others like interprocess communication, variables, process dependency management, etc. Let's Define a Simple Workflow we want to achieve DAG = Tasks1 >> Wait for a Day >> Task2 Above is the DAG, where two tasks are executed, second task depends on first task and is executed after a d