DataFlow in Expression Language

From QPR ProcessAnalyzer Wiki
Revision as of 17:10, 8 December 2022 by Ollvihe (talk | contribs)
Jump to navigation Jump to search

DataFlow is an object representing a stream of tabular data. DataFlow contains data with the similar structure as DataFrame, but difference is that in the DataFrame all its contents is stored to the system memory. If there is lot of data, also lot of memory is required when using DataFrames. On the other hand in the DataFlow, contents "flows" from the source to the destination, and data can be manipulated, while having only a small portion of the entire data in memory at the same time. Thus, DataFlows are suitable for ETL where data volumes are high.

DataFlow continues to run until it completes. DataFlow will complete automatically, when all queried items have been returned. DataFlow can also be completed explicitly by calling the Complete function. When the DataFlow has been completed, no new items can be added to it. When collecting the DataFlow to an in-memory DataFrame, the Collect call waits until the DataFlow completes, to make sure all items are included to the DataFrame.

Property Description
IsCompleted (boolean) Returns true when the Complete function has been called for the DataFlow and there are no more unread items in it.
Function Parameters Description
Append DataFrame to append

Appends given DataFrame to the DataFlow. Returns the DataFlow object.

Examples:

ToDataFlow(ToDataFrame([], ["id", "color"]))
  .Append(ToDataFrame([[1, "red"], [2, "green"]], ["id", "color"]))
  .Complete()
  .Collect()
  .ToCsv()
Collect Parameters (Dictionary)

Returns in-memory DataFrame extracted from the DataFlow. Returns the extracted data as in-memory DataFrame or null if either the timeout has been exceeded or the flow has been completed and is empty. Parameters:

  1. CollectChunk: Takes the next IDataFrame (#70613#) object found in the data flow and converts it into an in-memory DataFrame.
  2. Timeout: Maximum number of milliseconds to wait for data to appear into the DataFlow before exiting with null value as result.

Examples:

ToDataFlow(ToDataFrame([], ["id", "color"]))
  .Append(ToDataFrame([[1, "red"], [2, "green"]], ["id", "color"]))
  .Append(ToDataFrame([[3, "blue"]], ["id", "color"]))
  .Complete()
  .Collect()
  .ToCsv()
ToDataFlow(ToDataFrame([], ["id", "color"]))
  .Append(ToDataFrame([[1, "red"], [2, "green"]], ["id", "color"]))
  .Append(ToDataFrame([[3, "blue"]], ["id", "color"]))
  .Complete()
  .Collect(#{"CollectChunk": true})
  .ToCsv()
Complete (none)

Declares that the DataFlow is completed, i.e., there won't be any new items anymore added to the DataFlow.

Examples:

ToDataFlow(ToDataFrame([], ["id", "color"]))
  .Append(ToDataFrame([[1, "red"], [2, "green"]], ["id", "color"]))
  .Complete()
  .Collect()
  .ToCsv()
Persist (Datatable)
  1. Datatable name (String)
  2. Additional parameters (Dictionary)
Writes DataFlow into datatable. Works similarly as the same function in the DataFrame.

Create new DataFlow:

Function Parameters Description
Persist (Datatable)
  1. DataFrame

Creates new DataFlow and optionally initializes it with given DataFrame.

Examples:

ToDataFlow(ToDataFrame([[1, "red"], [2, "green"]], ["id", "color"]))
  .Append(ToDataFrame([[3, "blue"]], ["id", "color"]))
  .Complete()
  .Collect()
  .ToCsv()