Airflow

A tool to express and execute workflows as directed acyclic graphs (DAGs).

Created 3 years ago - Last update 3 years ago

Apache Airflow is a tool to express and execute workflows as directed acyclic graphs (DAGs). It includes utilities to schedule tasks, monitor task progress and handle task dependencies. This instance uses the Bitnami images

Deployed services

airflow-db

bitnami/postgresql:$$cap_postgresql_version

airflow-postgresql-persistance:/bitnami

always

airflow-redis

bitnami/redis:$$cap_redis_version

airflow-redis-persistance:/bitnami

always

airflow

bitnami/airflow:$$cap_airflow_version

$$cap_airflow_dag_path:/opt/bitnami/airflow/dags
$$cap_airflow_py_requirements:/bitnami/python/requirements.txt

8080

always

airflow-worker

bitnami/airflow-worker:$$cap_airflow_worker_version

airflow-data:/opt/bitnami/airflow/data
$$cap_airflow_dag_path:/opt/bitnami/airflow/dags
$$cap_airflow_py_requirements:/bitnami/python/requirements.txt

airflow-scheduler

bitnami/airflow-scheduler:$$cap_airflow_scheduler_version

$$cap_airflow_dag_path:/opt/bitnami/airflow/dags
$$cap_airflow_py_requirements:/bitnami/python/requirements.txt

always

Variables

Bitnami's Posgresql Version

Default: 12.12.0

$$cap_postgresql_version
Check the tag version on https://hub.docker.com/r/bitnami/postgresql

Postresql database name

Default: airflow_db

$$cap_postgresql_name
Name of the postgresql db

Postresql database user

Default: airflow_db_user

$$cap_postgresql_user
Password to the postgresql db

Postresql password

Default: $$cap_gen_random_hex(16)

$$cap_postgresql_password
Password to the postgresql db

Bitnami's redis Version

Default: 7.0.4

$$cap_redis_version
Check the tag version on https://hub.docker.com/r/bitnami/redis

Redis password

Default: $$cap_gen_random_hex(16)

$$cap_redis_password
Password to redis db

Bitnami's airflow Version

Default: 2.3.3

$$cap_airflow_version
Check the tag version on https://hub.docker.com/r/bitnami/airflow

Bitnami's airflow worker Version

Default: 2.3.3

$$cap_airflow_worker_version
Check the tag version on https://hub.docker.com/r/bitnami/airflow-worker

Bitnami's airflow scheduler Version

Default: 2.3.3

$$cap_airflow_scheduler_version
Check the tag version on https://hub.docker.com/r/bitnami/airflow-scheduler

Airflow user

Default: airflow_user

$$cap_airflow_user
Airflow's Admin user name

Airflow password

Default: $$cap_gen_random_hex(16)

$$cap_airflow_password
Airflow Admin's password

Airflow email

Default: undefined

$$cap_airflow_email
Email address that will receive notifications from airflow (need SMPT details)

Load Airflow DAG Examples?

Default: no

$$cap_airflow_examples
Wheter to load or not load sample DAGs (yes/no)

Airflow Secret Key

Default: $$cap_gen_random_hex(16)

$$cap_airflow_secret_key
Secret key used to run your flask app. It should be as random as possible.

DAGs Volume

Default: airflow-dags

$$cap_airflow_dag_path
Name of the DAG volume or Path to the custom DAGs folder, make sure the folder is accesible

Path or volume to a requirements.txt file

Default: airflow-requstxt

$$cap_airflow_py_requirements
Installation of additional python modules at start-up time, make sure the file is accesible

SMTP Host

Default: undefined

$$cap_smtp_host
Host for SMPT connection

SMTP Port

Default: undefined

$$cap_smtp_port
Port for SMPT connection

Use startTLS?

Default: no

$$cap_smtp_starttls
Do you want to use startTLS? (yes/no)

Use SSL?

Default: no

$$cap_smtp_ssl
Check the tag version on https://hub.docker.com/r/bitnami/airflow-scheduler

SMTP User

Default: undefined

$$cap_smtp_user
Check the tag version on https://hub.docker.com/r/bitnami/airflow-scheduler

SMTP Password

Default: undefined

$$cap_smtp_password
Check the tag version on https://hub.docker.com/r/bitnami/airflow-scheduler

Mail from address

Default: undefined

$$cap_smtp_mail_from
Specofy who the email is from