how to simply import custom module (file with sql queries)

Question

I have some SQL queries I want to store in a file so I can call them easily. The file is named sql_queries.py and stored in a helpers folder under plugins folder.

from airflow.plugins_manager import AirflowPlugin
import helpers

class SqlQueries:
    songplay_table_insert = ("""
        SELECT *
        FROM XXXX
    """)

class Project(AirflowPlugin):
    name = "Project"
    helpers = [SqlQueries]

I'm trying to import my queries from my dag like this from airflow.helpers import SqlQueries

I've used the same system for operators and it works. Why it doesn't in this case?

You can simply create a folder SQL in DAG folder and place your file there. Then import it as from SQL.SQLqueries import * . — Priya Agarwal, May 27 '20 at 17:11

kaxil · Accepted Answer · 2020-05-28T09:36:34.013

1

You do not need Airflow plugins for this, you can import any custom module in your DAG like a normal Python module.

I would say store sql_queries.py in any folder that is on PYTHONPATH and contains __init__.py

Example:

You can directly place sql_queries.py inside DAGs directory and import it as follows in your DAG as the dags/ directory is already in the PYTHONPATH:

import sql_queries

edited May 28 '20 at 09:36

answered May 27 '20 at 18:18

kaxil

17,706
2
59
78

What should I include in my init file in that case? – Simon Breton May 27 '20 at 22:18
1

`__init__.py` just needs to be a blank file. Having a blank `__init__.py` file inside any folder makes it a package. Check this SO post for more details https://stackoverflow.com/questions/448271/what-is-init-py-for – kaxil May 28 '20 at 09:36

score 0 · Answer 2 · answered May 27 '20 at 07:27

Because Airflow plugins don't support the creation of helpers. You can code custom executors, macros, admin_views, flask_blueprints, menu_links, appbuilder_views, appbuilder_menu_items, global_operator_extra_links and operator_extra_links.

You can have the script as a regular package import or convert that piece of code to a method within a custom Hook.

how to simply import custom module (file with sql queries)

2 Answers2