{"revision":"c10276e953456835cdbb1f3a60d58ae1980be353","document":{"id":21188,"path":"integrate-applications/integration-hub/data-stream-design.md","title":"Data Stream action design considerations","bundle":null,"doc_type":null,"release":"australia","body":"\n# Data Stream action design considerations\n\nFollow these considerations when creating a Data Stream action and when adding a Data Stream action to a flow to prevent errors and performance issues.\n\n## Considerations when creating a Data Stream action\n\nCreate Data Stream actions with these considerations in mind.\n\n-   **Write pagination logic according to third-party requirements**\n\n    Evaluate and understand the format required by your third-party endpoint. For example, you may need to write a script that sets the built-in `getNextPage` variable to true as long as there is a `nextPage` token in the response. If the response does not contain the token, then set the variable to false. Access variables in script using bracket or dot notation. For example, `variables['getNextPage']`. This variable only accepts the Boolean data type. The default value is false.\n\n-   **Ensure that preprocessing scripts do not call a MID Server or pause the action.**\n\n    Data Stream actions require a constant connection to the response stream. They do not support preprocessing scripts that pause the action to call a MID Server or pause the action to wait for a condition or duration. A preprocessing script that pauses the action for any reason returns an error message.\n\n-   **Convert pagination variable data types to perform math operations**\n\n    Pagination variables only support the string data type. To perform math operations, you must convert the value to an integer, perform any required operations, then convert it back to a string.\n\n    ```\n    variables['offset'] = (parseInt(variables['offset']) + parseInt(variables['limit'])).toString();\n    ```\n\n-   **Ensure that the pagination script has an end condition**\n\n    Avoid infinite loops in pagination requests by creating a condition that sets the `getNextPage` variable to false. Cancel any long-running flows. Always test Data Stream actions before using them in production.\n\n-   **Clear configuration page options carefully**\n\n    Clearing an option in a configuration page removes the step from the Data Stream outline and deletes all data associated with the step.\n\n-   **Consider the size of each page returned**\n\n    When configuring pagination options, try to ensure that the size of each page returned is under one GB. This ensures that flow designers who use the action won't encounter size limit issues when retrieving data from a MID Server. If the action returns more than one GB of data per page, the flow designer may need to increase the attachment size limit. For more information, see [MID Server support for Data Stream actions](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/integrate-applications/integration-hub/dsa-on-mid.md).\n\n\n**Note:**\n\n-   Nesting Data Stream actions is not supported.\n-   When using the retry policy, ensure that you are aware of these considerations:\n    -   Only a fixed time interval strategy is honoured for the data stream actions.\n    -   The total delay \\(count multiplied by the interval\\) can't exceed 30 seconds. For example, if you want retry for a maximum of 3 times, the maximum delay for each retry is 10 seconds.\n    -   Data stream step doesn’t go into the waiting state during retry.\n\n## Considerations when adding a Data Stream action to a flow\n\nAdd a Data Stream action to a flow with these considerations in mind.\n\n-   **Do not add actions with wait conditions to a Data Stream For Each logic block**\n\n    You cannot add an action that pauses the flow to a Data Stream For Each flow logic block. Data Stream actions require a constant connection to the response stream. For example, you cannot use the Ask for Approval or Wait for Condition actions within Data Stream For Each flow logic.\n\n    Some actions and steps always pause a flow and cannot be added to Data Stream For Each flow logic blocks. However, some actions and steps may pause the flow depending on how they are configured. Check the action configuration to ensure it does not pause the flow at runtime.\n\n<table id=\"table_mwj_dgw_3jb\"><tbody><tr><td rowspan=\"4\">\n\n**Always pauses the flow \\(unsupported with Data Stream actions\\)**\n\n</td><td>\n\nAsk for Approval action\n\n</td></tr><tr><td>\n\nAsk for Approval step\n\n</td></tr><tr><td>\n\nWait for Condition action\n\n</td></tr><tr><td>\n\nWait for Condition step\n\n</td></tr><tr><td rowspan=\"3\">\n\n**May pause the flow depending on configuration**\n\n</td><td>\n\nCreate Catalog Task action\n\n</td></tr><tr><td>\n\nCreate Task action\n\n</td></tr><tr><td>\n\nCreate Task step\n\n</td></tr></tbody>\n</table>-   **Do not add actions that run on the MID Server to a Data Stream For Each logic block**\n\n    Some actions and steps always run on a MID Server and cannot be added to Data Stream For Each flow logic blocks. However, some actions and steps may run on a MID Server depending on how they are configured. Check the action configuration to ensure it does not run on the MID Server at runtime.\n\n<table id=\"table_r4b_thw_3jb\"><tbody><tr><td rowspan=\"3\">\n\n**Always runs on a MID Server \\(unsupported with Data Stream actions\\)**\n\n</td><td>\n\nJDBC step\n\n</td></tr><tr><td>\n\nPowerShell step\n\n</td></tr><tr><td>\n\nSSH step\n\n</td></tr><tr><td rowspan=\"3\">\n\n**May run on a MID Server depending on configuration**\n\n</td><td>\n\nREST step\n\n</td></tr><tr><td>\n\nScript step\n\n</td></tr><tr><td>\n\nSOAP step\n\n</td></tr></tbody>\n</table>-   **Turn off reporting when consuming a large number of records in the For Each logic block**\n\n    Iterating over a large number of records can be resource intensive, especially when the For Each logic block includes complex actions for each iteration. To avoid performance issues, turn off reporting using the **com.snc.process\\_flow.reporting.level** system property. For more information, see [Workflow Studio flow system properties](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/build-workflows/flow-designer-system-properties.md).\n\n\n**Parent Topic:**[Data Stream actions and pagination](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/integrate-applications/integration-hub/data-stream-actions.md)\n\n"}}