Skip to content

[Feature] show delete and insert times on incremental logs #12189

@alejandrofm

Description

@alejandrofm

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Add, delete, and insert times on incremental DBT logs, currently showing:

INFO - 17:24:45 Running with dbt=1.10.13
INFO - 17:24:45 Registered adapter: redshift=1.9.5
INFO - 17:24:46 Found 7 models, 17 sources, 524 macros
INFO - 17:24:46
INFO - 17:24:46 Concurrency: 4 threads (target='dev')
INFO - 17:24:46
INFO - 17:24:47 1 of 1 START sql table model collections_stage.sc_pmt_cb_information ........... [RUN]
INFO - 17:27:32 1 of 1 OK created sql table model collections_stage.sc_pmt_cb_information ...... [SUCCESS in 165.19s]
INFO - 17:27:32
INFO - 17:27:32 Finished running 1 table model in 0 hours 2 minutes and 46.52 seconds (166.52s).
INFO - 17:27:32
INFO - 17:27:32 Completed successfully
INFO - 17:27:32
INFO - 17:27:32 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

Describe alternatives you've considered

proposal:
ADDING DELETING and INSERTING on logs, like this:

INFO - 17:24:45 Running with dbt=1.10.13
INFO - 17:24:45 Registered adapter: redshift=1.9.5
INFO - 17:24:46 Found 7 models, 17 sources, 524 macros
INFO - 17:24:46
INFO - 17:24:46 Concurrency: 4 threads (target='dev')
INFO - 17:24:46
INFO - 17:24:47 1 of 1 START sql table model collections_stage.sc_pmt_cb_information ........... [RUN]
INFO - 17:24:47 1 of 1 START INCREMENTAL delete table model collections_stage.sc_pmt_cb_information ........... [RUN]
INFO - 17:27:32 1 of 1 INCREMENTAL delete OK on table model collections_stage.sc_pmt_cb_information ...... [SUCCESS in 165.19s]
INFO - 17:24:47 1 of 1 START INSERT table model collections_stage.sc_pmt_cb_information ........... [RUN]
INFO - 17:27:32 1 of 1 INSERT OK on table model collections_stage.sc_pmt_cb_information ...... [SUCCESS in 165.19s]

INFO - 17:27:32 1 of 1 MODEL OK created sql table model collections_stage.sc_pmt_cb_information ...... [SUCCESS in 165.19s]
INFO - 17:27:32
INFO - 17:27:32 Finished running 1 table model in 0 hours 2 minutes and 46.52 seconds (166.52s).
INFO - 17:27:32
INFO - 17:27:32 Completed successfully
INFO - 17:27:32
INFO - 17:27:32 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

Who will this benefit?

It will help debug long-running DBTs, on databases that are slow to delete records and/or identify ways to improve the DBT, looking only at the main logs.
Having a ton of DBTs makes it hard to go through detailed logs.

Are you interested in contributing this feature?

No response

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions