fix(state window): status is null if window data start with null

[Tony2h]

Description

Please briefly describe the code changes in this pull request.

Jira: https://jira.taosdata.com:18080/browse/TD-

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @Tony2h, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves an issue where state window aggregations produced incorrect NULL results when the window data began with NULL values. It achieves this by introducing a dedicated flag for window operations and refining the aggregation logic to properly identify non-NULL values within such windows. The changes are thoroughly validated with new and updated test cases, ensuring the reliability and correctness of state window functionality, including the new extend parameter.

Highlights

• New hasWindow Flag: A new boolean field hasWindow has been introduced in SqlFunctionCtx and SExprSupp structures. This flag specifically denotes if a function is used with a time window, providing a more granular distinction than the existing hasWindowOrGroup flag.
• Improved groupConstValueFunction Logic: The groupConstValueFunction has been updated to correctly handle scenarios where a state window starts with NULL values. Previously, this could lead to the entire window's status being NULL. The revised logic now intelligently searches for the first non-NULL value within a state window, or defaults to the first row for group by operations, ensuring accurate aggregation results.
• Window Operator Initialization Updates: Various window operators, including countwindowoperator.c, eventwindowoperator.c, externalwindowoperator.c, and timewindowoperator.c, have been modified to correctly set the new hasWindow flag to true during their initialization. The groupoperator.c explicitly sets hasWindow to false.
• Refactored and Expanded Test Cases: Existing test cases for state windows have been refactored, including renaming and moving test_state_window.py. A new test case, test_state_window_start_with_null, has been added to validate the fix for NULL values at the start of state windows. Additionally, a comprehensive new test file, test_state_window_extend.py, has been introduced to cover the functionality of the extend parameter in state window operations across various table types and stream computing scenarios.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request fixes an issue where the state of a state_window could be incorrectly determined as NULL if the window's data starts with a NULL value. The fix introduces a new hasWindow flag to differentiate window operations from group by operations. The core logic in groupConstValueFunction is updated to re-evaluate the state for a window if the previously determined state was NULL.

The changes are logical and the new test case validates the fix for the described scenario. However, I've found a potential correctness issue in the new logic for finding a non-null state value within a data chunk, which I've detailed in a specific comment. The current implementation only checks the first and last rows, which could fail for certain data patterns. My suggestion is to scan for the first available non-null value to make the logic more robust.

Additionally, the pull request includes a fix for a memory leak in tDeserializeSCMCreateStreamReqImpl, which is a good improvement.

[gemini-code-assist]

The current logic to find a non-null value for a window only checks the start and end rows of the data chunk. This is a performance heuristic but it's not correct in all cases. For a state_window, the state is constant across the window (ignoring NULLs). If a block of data for a window has non-null values only in the middle, and NULL at both start and end (e.g., [NULL, 'a', 'a', NULL]), this logic will incorrectly determine the state is NULL.

To ensure correctness, the code should scan the chunk to find the first non-null value. This will be more robust.

    if (isWindow) {
      // for 'window', find first non-null value in the current data chunk
      for (int32_t i = 0; i < pInput->numOfRows; ++i) {
        int32_t currentRow = pInput->startRowIndex + i;
        if (!colDataIsNull_s(pInputCol, currentRow)) {
          valueRowIndex = currentRow;
          break;
        }
      }
    }