feat(cli): support project paths and options from environment variables

docs/integrations/github.md

@@ -59,6 +59,21 @@ jobs:
 
 Next checkout the [Core Bot Behavior Configuration Guide](#core-bot-behavior-configuration-guide) to see how to configure the bot's core behavior. Then checkout [Bot Configuration](#bot-configuration) to see how to configure the bot's behavior in general. Finally, checkout [Custom Workflow Configuration](#custom-workflow-configuration) to see the full set of customizations available to the bot.
 
+### Project paths
+
+The bot shares the `SQLMESH_PATHS` environment variable with the `sqlmesh` CLI. You can use it instead of `-p` in a workflow step:
+
+```yaml
+- name: Run CI/CD Bot
+  env:
+    SQLMESH_PATHS: ${{ github.workspace }}
+  run: sqlmesh_cicd github --token ${{ secrets.GITHUB_TOKEN }} run-all
+```
+
+An explicit `-p` or `--paths` option replaces the entire environment path list. An unset or empty `SQLMESH_PATHS` uses the existing default project path. Export the variable in the workflow environment before invoking the bot; a project's `.env` file cannot locate the project that contains it.
+
+Spaces and Unicode characters remain part of each path. For multiple projects, separate paths with `:` on POSIX runners or `;` on Windows runners. Empty path-list segments refer to the current directory. Use repeated `-p` options for directory names that contain the platform's path separator. See the [CLI reference](../reference/cli.md#project-paths) for examples. Automatic command-option variables such as `SQLMESH_PLAN_START` apply to the main `sqlmesh` CLI; the bot shares the explicit `SQLMESH_PATHS` binding.
+
 ## Core Bot Behavior Configuration Guide
 
 There are two fundamental ways the bot can be configured: synchronized or desynchronized production code and data.

docs/reference/cli.md

@@ -7,7 +7,7 @@ Usage: sqlmesh [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --version            Show the version and exit.
-  -p, --paths TEXT     Path(s) to the SQLMesh config/project.
+  -p, --paths PATH     Path(s) to the SQLMesh config/project.
   --config TEXT        Name of the config object. Only applicable to
                        configuration defined using Python script.
   --gateway TEXT       The name of the gateway.
@@ -52,6 +52,69 @@ Commands:
   lint                    Run the linter for the target model(s).
 ```
 
+## Environment variables
+
+CLI options accept defaults from environment variables. Use `SQLMESH_` followed by the command names and the option's Click parameter name, all uppercase with hyphens replaced by underscores. Include every command name for nested commands. The parameter name usually matches the long option, but some differ, as shown below. Aliases share the same environment variable.
+
+| Environment variable | CLI option |
+| --- | --- |
+| `SQLMESH_PATHS` | `sqlmesh --paths` |
+| `SQLMESH_CONFIG` | `sqlmesh --config` |
+| `SQLMESH_PLAN_START` | `sqlmesh plan --start` |
+| `SQLMESH_PLAN_SKIP_BACKFILL` | `sqlmesh plan --skip-backfill` or `sqlmesh plan --dry-run` |
+| `SQLMESH_AUDIT_MODELS` | `sqlmesh audit --model` |
+| `SQLMESH_CREATE_TEST_QUERIES` | `sqlmesh create_test --query` |
+| `SQLMESH_CREATE_TEST_VARIABLES` | `sqlmesh create_test --var` |
+| `SQLMESH_TABLE_DIFF_LIMIT` | `sqlmesh table_diff --limit` |
+| `SQLMESH_STATE_EXPORT_OUTPUT_FILE` | `sqlmesh state export --output-file` |
+| `SQLMESH_TEST_LOCAL` | `sqlmesh test --local` |
+| `SQLMESH_LINT_LOCAL` | `sqlmesh lint --local` |
+
+Options with an explicit environment variable retain their names: `SQLMESH_GATEWAY`, `SQLMESH_IGNORE_WARNINGS`, and `SQLMESH_DOTENV_PATH`. Environment variables do not supply positional arguments, such as the environment in `sqlmesh plan dev`.
+
+An explicit CLI option takes precedence over its environment variable. For example, `sqlmesh plan --start 2025-02-01` uses that date even if `SQLMESH_PLAN_START` specifies another date. Values use the same type validation as the corresponding CLI option.
+
+### Project paths
+
+Set `SQLMESH_PATHS` to select a project when running SQLMesh from another directory:
+
+```bash
+export SQLMESH_PATHS="/work/analytics project"
+sqlmesh info
+```
+
+Spaces and Unicode characters remain part of each path. For multiple projects, separate paths with `:` on POSIX or `;` on Windows. For example, on POSIX:
+
+```bash
+export SQLMESH_PATHS="/work/analytics project:/work/finance α"
+sqlmesh info
+```
+
+In PowerShell, a single project path can be set with:
+
+```powershell
+$env:SQLMESH_PATHS = 'C:\work\analytics project'
+sqlmesh info
+```
+
+Windows path lists use semicolons, preserving drive-letter colons. Shell quotes delimit the assignment and are not part of the value.
+
+Explicit `-p` or `--paths` options replace the entire environment path list. Repeat the option to select multiple projects:
+
+```bash
+sqlmesh -p "/work/another project" -p "/work/finance α" info
+```
+
+An unset or empty `SQLMESH_PATHS` uses the existing default project path. Relative paths resolve from the invocation directory. Avoid leading, trailing, or repeated path separators: empty segments are retained and refer to the current directory. If a directory name contains the platform's path separator, pass it with an explicit `-p` option.
+
+### Dotenv timing and precedence
+
+Root options, including project paths, are parsed before SQLMesh loads project `.env` files or the file selected by `--dotenv`. Export `SQLMESH_PATHS` before invoking SQLMesh. A project `.env` file cannot select the project that SQLMesh must find to load that file. Other root options, such as `SQLMESH_CONFIG` and `SQLMESH_DOTENV_PATH`, also need to be present before invocation.
+
+For commands that load project configuration, subcommand options are parsed after dotenv loading. Variables such as `SQLMESH_PLAN_START`, `SQLMESH_TEST_LOCAL`, and `SQLMESH_LINT_LOCAL` can therefore come from the selected project's `.env` file. SQLMesh loads dotenv values with replacement enabled, so they can override values already in the process environment. Explicit CLI options still take precedence.
+
+CLI option variables are separate from the `SQLMESH__...` variables that configure project settings. See the [configuration reference](./configuration.md#environment-variables).
+
 ## audit
 
 ```

docs/reference/configuration.md

@@ -4,6 +4,14 @@ This page lists SQLMesh configuration options and their parameters. Learn more a
 
 Configuration options for model definitions are listed in the [model configuration reference page](./model_configuration.md).
 
+## Environment variables
+
+Project settings use environment variables beginning with `SQLMESH__`, with double underscores separating configuration fields. For example, `SQLMESH__MODEL_DEFAULTS__DIALECT=snowflake` sets the default model dialect, and `SQLMESH__DEFAULT_GATEWAY=production` sets the project's default gateway. These variables override values in configuration files. See the [configuration guide](../guides/configuration.md#environment-variables) for details.
+
+CLI option defaults use a single underscore after `SQLMESH`, such as `SQLMESH_PATHS`, `SQLMESH_CONFIG`, or `SQLMESH_PLAN_START`. An explicit CLI option takes precedence over its environment variable. For example, `SQLMESH_GATEWAY` selects the gateway for a command, while `SQLMESH__DEFAULT_GATEWAY` configures the project's default gateway.
+
+Export root CLI option variables before invoking SQLMesh. Root options are parsed before project `.env` files or a custom `--dotenv` file are loaded, so `SQLMESH_PATHS` inside a project `.env` cannot locate that project. Subcommand option variables can come from the loaded dotenv file, whose values replace existing process environment values. See the [CLI environment-variable reference](./cli.md#environment-variables) for naming, path lists, and precedence.
+
 ## Root configurations
 
 A SQLMesh project configuration consists of root level parameters within which other parameters are defined.

sqlmesh/cli/main.py

@@ -66,7 +66,11 @@ def _sqlmesh_version() -> str:
         return "0.0.0"
 
 
-@click.group(cls=_SQLMeshGroup, no_args_is_help=True)
+@click.group(
+    cls=_SQLMeshGroup,
+    no_args_is_help=True,
+    context_settings={"auto_envvar_prefix": "SQLMESH"},
+)
 @click.version_option(version=_sqlmesh_version(), message="%(version)s")
 @opt.paths
 @opt.config
@@ -131,13 +135,6 @@ def cli(
     load = True
     # Local-only gating must hold for any number of --paths, so it stays outside the block below.
     load_state = ctx.invoked_subcommand not in LOCAL_ONLY_COMMANDS
-    # The parent callback constructs Context before Click invokes the subcommand, so inspect its
-    # parsed args here.
-    if (
-        ctx.invoked_subcommand in OPTIONAL_LOCAL_COMMANDS
-        and "--local" in ctx.meta["subcommand_args"]
-    ):
-        load_state = False
 
     if len(paths) == 1:
         path = os.path.abspath(paths[0])
@@ -152,6 +149,19 @@ def cli(
         load = False
 
     configs = load_configs(config, Context.CONFIG_TYPE, paths, dotenv_path=dotenv)
+    # The subcommand is parsed after this callback. Resolve its local flag after dotenv loading
+    # so Context avoids state access for the same effective value the subcommand will receive.
+    if ctx.invoked_subcommand and ctx.invoked_subcommand in OPTIONAL_LOCAL_COMMANDS:
+        if "--local" in ctx.meta["subcommand_args"]:
+            load_state = False
+        else:
+            command = t.cast(click.Group, ctx.command).get_command(ctx, ctx.invoked_subcommand)
+            assert command is not None
+            with click.Context(command, info_name=ctx.invoked_subcommand, parent=ctx) as child:
+                local = next(param for param in command.params if param.name == "local")
+                value = local.value_from_envvar(child)
+                if value is not None:
+                    load_state = not local.type_cast_value(child, value)
     log_limit = list(configs.values())[0].log_limit
 
     remove_excess_logs(log_file_dir, log_limit)

sqlmesh/cli/options.py

@@ -8,8 +8,10 @@
 paths = click.option(
     "-p",
     "--paths",
+    type=click.Path(readable=False),
     multiple=True,
     default=[os.getcwd()],
+    envvar="SQLMESH_PATHS",
     help="Path(s) to the SQLMesh config/project.",
 )
 

tests/cli/test_cli.py

@@ -12,6 +12,7 @@
 from click.testing import CliRunner
 from sqlglot import __version__ as SQLGLOT_VERSION
 from sqlmesh import RuntimeEnv
+from sqlmesh.cli import main as cli_main
 from sqlmesh._version import __version__ as SQLMESH_VERSION
 from sqlmesh.cli.project_init import ProjectTemplate, init_example_project
 from sqlmesh.cli.main import cli
@@ -2582,6 +2583,187 @@ def _setup_local_only_project(tmp_path, mocker):
     return _patch_state_access(mocker)
 
 
+@pytest.fixture
+def cli_env(monkeypatch):
+    for name in (
+        "SQLMESH_PATHS",
+        "SQLMESH_CONFIG",
+        "SQLMESH_TEST_LOCAL",
+        "SQLMESH_LINT_LOCAL",
+        "SQLMESH_DOTENV_PATH",
+        "SQLMESH__MODEL_DEFAULTS__DIALECT",
+    ):
+        # Register an undo even when absent, since dotenv can add the variable directly.
+        monkeypatch.setenv(name, "")
+        monkeypatch.delenv(name)
+
+
+@pytest.mark.parametrize("name", ["project", "analytics project 分析"])
+@pytest.mark.parametrize("relative", [False, True])
+def test_paths_env_selects_project_from_another_directory(
+    runner, tmp_path, monkeypatch, cli_env, name, relative
+):
+    project = tmp_path / name
+    _create_local_only_project(project, "env_project")
+    elsewhere = tmp_path / "elsewhere"
+    elsewhere.mkdir()
+    monkeypatch.chdir(elsewhere)
+    monkeypatch.setenv("SQLMESH_PATHS", os.path.relpath(project) if relative else str(project))
+    result = runner.invoke(cli, ["--log-file-dir", str(tmp_path), "info"])
+    assert result.exit_code == 0, result.output
+    assert "Models: 1" in result.output
+
+
+@pytest.mark.parametrize("command", ["test", "lint"])
+@pytest.mark.parametrize("value", ["true", "1", "false", "0", "", "invalid"])
+@pytest.mark.parametrize("explicit", [False, True])
+def test_local_env_state_gating(
+    runner, tmp_path, mocker, monkeypatch, cli_env, command, value, explicit
+):
+    state = _setup_local_only_project(tmp_path, mocker)
+    init = mocker.spy(Context, "__init__")
+    monkeypatch.setenv(f"SQLMESH_{command.upper()}_LOCAL", value)
+    args = ["--paths", str(tmp_path), command]
+    if explicit:
+        args.append("--local")
+    result = runner.invoke(cli, args)
+    if value == "invalid" and not explicit:
+        assert result.exit_code == 2, result.output
+        assert "valid boolean" in result.output
+        init.assert_not_called()
+        state.assert_not_called()
+    elif explicit or value in ("true", "1"):
+        assert result.exit_code == 0, result.output
+        assert init.call_args.kwargs["load_state"] is False
+        state.assert_not_called()
+    else:
+        assert init.call_args.kwargs["load_state"] is True
+        state.assert_called()
+
+
+@pytest.mark.parametrize("command", ["test", "lint"])
+@pytest.mark.parametrize("custom_dotenv", [False, True])
+@pytest.mark.parametrize("value", ["true", "false"])
+def test_local_env_from_dotenv(
+    runner, tmp_path, mocker, monkeypatch, cli_env, command, custom_dotenv, value
+):
+    state = _setup_local_only_project(tmp_path, mocker)
+    init = mocker.spy(Context, "__init__")
+    variable = f"SQLMESH_{command.upper()}_LOCAL"
+    # The existing config loader lets dotenv override the process environment.
+    monkeypatch.setenv(variable, "false" if value == "true" else "true")
+    env_file = tmp_path / ("custom.env" if custom_dotenv else ".env")
+    env_file.write_text(f"{variable}={value}\n", encoding="utf-8")
+    args = ["--paths", str(tmp_path)]
+    if custom_dotenv:
+        args.extend(["--dotenv", str(env_file)])
+    result = runner.invoke(cli, [*args, command])
+    assert init.call_args.kwargs["load_state"] is (value == "false")
+    if value == "true":
+        assert result.exit_code == 0, result.output
+        state.assert_not_called()
+    else:
+        state.assert_called()
+
+
+@pytest.mark.parametrize("command", ["test", "lint", "format"])
+def test_paths_env_multiple_projects_local(runner, tmp_path, mocker, monkeypatch, cli_env, command):
+    projects = (tmp_path / "first project", tmp_path / "second 分析")
+    for index, project in enumerate(projects):
+        _create_local_only_project(project, f"project_{index}")
+    state = _patch_state_access(mocker)
+    monkeypatch.setenv("SQLMESH_PATHS", os.pathsep.join(str(project) for project in projects))
+    if command != "format":
+        monkeypatch.setenv(f"SQLMESH_{command.upper()}_LOCAL", "true")
+    result = runner.invoke(cli, [command])
+    assert result.exit_code == 0, result.output
+    state.assert_not_called()
+
+
+def test_paths_env_init_new_destination(runner, tmp_path, monkeypatch, cli_env):
+    project = tmp_path / "new project"
+    project.mkdir()
+    monkeypatch.setenv("SQLMESH_PATHS", str(project))
+    result = runner.invoke(cli, ["init", "duckdb", "--template", "empty"])
+    assert result.exit_code == 0, result.output
+    assert (project / "config.yaml").exists()
+
+
+@pytest.mark.parametrize("invalid_config", [False, True])
+def test_paths_env_project_errors(runner, tmp_path, monkeypatch, cli_env, invalid_config):
+    project = tmp_path / "project"
+    if invalid_config:
+        project.mkdir()
+        (project / "config.yaml").write_text("not a config\n", encoding="utf-8")
+    monkeypatch.setenv("SQLMESH_PATHS", str(project))
+    result = runner.invoke(cli, ["info"])
+    assert result.exit_code == 1, result.output
+    assert (
+        "Invalid YAML configuration" if invalid_config else "project config could not be found"
+    ) in result.output
+
+
+def test_paths_env_in_dotenv_does_not_change_root_paths(runner, tmp_path, monkeypatch, cli_env):
+    project = tmp_path / "project"
+    alternate = tmp_path / "alternate"
+    _create_local_only_project(project, "selected_project")
+    _create_local_only_project(alternate, "alternate_project")
+    (alternate / "models" / "second.sql").write_text(
+        "MODEL(name alternate_project.second); SELECT 2 AS col", encoding="utf-8"
+    )
+    (project / ".env").write_text(f"SQLMESH_PATHS='{alternate}'\n", encoding="utf-8")
+    monkeypatch.setenv("SQLMESH_PATHS", str(project))
+    result = runner.invoke(cli, ["info"])
+    assert result.exit_code == 0, result.output
+    assert "Models: 1" in result.output
+
+
+def test_paths_env_only_in_custom_dotenv_does_not_select_project(
+    runner, tmp_path, monkeypatch, mocker, cli_env
+):
+    project = tmp_path / "project"
+    _create_local_only_project(project, "env_project")
+    env_file = tmp_path / "custom.env"
+    env_file.write_text(f"SQLMESH_PATHS='{project}'\n", encoding="utf-8")
+    load_configs = mocker.spy(cli_main, "load_configs")
+    runner.invoke(cli, ["--dotenv", str(env_file), "info"])
+    paths = next(param for param in cli.params if param.name == "paths")
+    assert load_configs.call_args.args[2] == tuple(paths.default)
+
+
+@pytest.mark.parametrize("empty_segment", [False, True])
+def test_paths_env_preserves_loader_expansion(
+    runner, tmp_path, monkeypatch, mocker, cli_env, empty_segment
+):
+    projects = [tmp_path / "projects" / "first", tmp_path / "projects" / "second"]
+    for index, project in enumerate(projects):
+        _create_local_only_project(project, f"env_project_{index}")
+    if empty_segment:
+        invocation = tmp_path / "invocation"
+        _create_local_only_project(invocation, "invocation")
+        monkeypatch.chdir(invocation)
+        env_value = os.pathsep.join([str(projects[0]), "", str(projects[1])])
+        expected = {projects[0], invocation, projects[1]}
+    else:
+        env_value = str(tmp_path / "projects" / "*")
+        expected = set(projects)
+    monkeypatch.setenv("SQLMESH_PATHS", env_value)
+    init = mocker.spy(Context, "__init__")
+    result = runner.invoke(cli, ["format"])
+    assert result.exit_code == 0, result.output
+    assert set(init.call_args.kwargs["config"]) == expected
+
+
+def test_cli_env_separate_from_project_config(runner, tmp_path, monkeypatch, mocker, cli_env):
+    _create_local_only_project(tmp_path, "env_project")
+    monkeypatch.setenv("SQLMESH_PATHS", str(tmp_path))
+    monkeypatch.setenv("SQLMESH__MODEL_DEFAULTS__DIALECT", "spark")
+    init = mocker.spy(Context, "__init__")
+    result = runner.invoke(cli, ["info"])
+    assert result.exit_code == 0, result.output
+    assert next(iter(init.call_args.kwargs["config"].values())).model_defaults.dialect == "spark"
+
+
 def test_format_runs_without_state(runner: CliRunner, tmp_path: Path, mocker):
     mock = _setup_local_only_project(tmp_path, mocker)
     result = runner.invoke(cli, ["--paths", str(tmp_path), "format"])