Anonymize data - Revision history

Ollvihe: /* Adding random noise */

2025-09-11T10:00:16Z

Adding random noise

← Older revision		Revision as of 10:00, 11 September 2025
Line 179:		Line 179:

	=== Adding random noise ===		=== Adding random noise ===
	This script ~~add~~ random noise to numerical data. The random noise is defined as a number between minimum and maximum value.		This script adds random noise to numerical data. The random noise is defined as a number between minimum and maximum value.
	<syntaxhighlight lang="typescript">		<syntaxhighlight lang="typescript">
	function AddRandomNoise(df, cols, min, max) {		function AddRandomNoise(df, cols, min, max) {

Ollvihe: /* Pseudonymization with running numbering */

2023-03-23T14:06:41Z

Pseudonymization with running numbering

← Older revision		Revision as of 14:06, 23 March 2023
Line 21:		Line 21:
	Following example script anonymizes selected columns in a datatable and writes the result to a new datatable. Each anonymized data value gets a distinct numeric value starting from one. Note that this approach may give different anonymized values, when the anonymization is done in the next time.		Following example script anonymizes selected columns in a datatable and writes the result to a new datatable. Each anonymized data value gets a distinct numeric value starting from one. Note that this approach may give different anonymized values, when the anonymization is done in the next time.
	<syntaxhighlight lang="typescript">		<syntaxhighlight lang="typescript">
	let skipAnonymizationForValues = [null, "Unknown"]; // list values that are not ~~anymized~~		let skipAnonymizationForValues = [null, "Unknown"]; // list values that are not anonymized
	let mappings = #{};		let mappings = #{};
	function PseudonymizeColumn(columnName, originalValue) {		function PseudonymizeColumn(columnName, originalValue) {

Ollvihe: /* Pseudonymization with running numbering */

2023-03-23T13:50:16Z

Pseudonymization with running numbering

← Older revision		Revision as of 13:50, 23 March 2023
Line 21:		Line 21:
	Following example script anonymizes selected columns in a datatable and writes the result to a new datatable. Each anonymized data value gets a distinct numeric value starting from one. Note that this approach may give different anonymized values, when the anonymization is done in the next time.		Following example script anonymizes selected columns in a datatable and writes the result to a new datatable. Each anonymized data value gets a distinct numeric value starting from one. Note that this approach may give different anonymized values, when the anonymization is done in the next time.
	<syntaxhighlight lang="typescript">		<syntaxhighlight lang="typescript">
			let skipAnonymizationForValues = [null, "Unknown"]; // list values that are not anymized
	let mappings = #{};		let mappings = #{};
	function PseudonymizeColumn(columnName, originalValue) {		function PseudonymizeColumn(columnName, originalValue) {
			if (IndexOf(skipAnonymizationForValues, originalValue) > -1) {
			return originalValue;
			}
	let dict;		let dict;
	if (!mappings.ContainsKey(columnName)) {		if (!mappings.ContainsKey(columnName)) {

Ollvihe: /* Pseudonymization with running numbering */

2023-03-21T15:45:30Z

Pseudonymization with running numbering

← Older revision		Revision as of 15:45, 21 March 2023
Line 29:		Line 29:
	} else {		} else {
	dict = mappings[columnName];		dict = mappings[columnName];
			}
			if (originalValue == null) {
			originalValue = "(null)";
	}		}
	if (!dict.ContainsKey(originalValue)) {		if (!dict.ContainsKey(originalValue)) {

Ollvihe: /* Adding random noise */

2023-02-05T23:21:05Z

Adding random noise

← Older revision		Revision as of 23:21, 5 February 2023
Line 186:		Line 186:
	AddRandomNoise(		AddRandomNoise(
	sourceDatatable.SqlDataFrame.Collect(),		sourceDatatable.SqlDataFrame.Collect(),
	["~~Item_OrderQuantity~~"],		["Cost"],
	100,		100,
	50		50

Ollvihe: /* Generalization: broader categories */

2023-02-05T23:20:01Z

Generalization: broader categories

← Older revision		Revision as of 23:20, 5 February 2023
Line 148:		Line 148:
	"North America": ["USA", "Canada", "Mexico"]		"North America": ["USA", "Canada", "Mexico"]
	}		}
	).Persist(sourceDatatable.Name + "_generalizedCategories", #{"ProjectId": sourceDatatable.Project.Id})		).Persist(sourceDatatable.Name + "_generalizedCategories", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Ollvihe at 00:03, 5 February 2023

2023-02-05T00:03:37Z

← Older revision		Revision as of 00:03, 5 February 2023
Line 49:		Line 49:
	sourceDatatable.SqlDataFrame.Collect(),		sourceDatatable.SqlDataFrame.Collect(),
	["Case Id", "Company Code", "Customer Name"]		["Case Id", "Company Code", "Customer Name"]
	).Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id});		).Persist(sourceDatatable.Name + "_pseudonymized", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Line 70:		Line 70:
	sourceDatatable.SqlDataFrame.Collect(),		sourceDatatable.SqlDataFrame.Collect(),
	["Case Id", "Company Code", "Customer Name"]		["Case Id", "Company Code", "Customer Name"]
	).Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id});		).Persist(sourceDatatable.Name + "_pseudonymizedWithHash", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Line 91:		Line 91:
	sourceDatatable.SqlDataFrame.Collect(),		sourceDatatable.SqlDataFrame.Collect(),
	["Case Id", "Company Code", "Customer Name"]		["Case Id", "Company Code", "Customer Name"]
	).Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id});		).Persist(sourceDatatable.Name + "_shuffled", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Line 113:		Line 113:
	["Case Id", "Company Code", "Customer Name"],		["Case Id", "Company Code", "Customer Name"],
	5		5
	).Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id});		).Persist(sourceDatatable.Name + "_masked", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Line 148:		Line 148:
	"North America": ["USA", "Canada", "Mexico"]		"North America": ["USA", "Canada", "Mexico"]
	}		}
	).Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id})		).Persist(sourceDatatable.Name + "_generalizedCategories", #{"ProjectId": sourceDatatable.Project.Id})
	</syntaxhighlight>		</syntaxhighlight>

Line 168:		Line 168:
	["Cost"],		["Cost"],
	100		100
	).Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id});		).Persist(sourceDatatable.Name + "_rounded", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Line 174:		Line 174:
	This script add random noise to numerical data. The random noise is defined as a number between minimum and maximum value.		This script add random noise to numerical data. The random noise is defined as a number between minimum and maximum value.
	<syntaxhighlight lang="typescript">		<syntaxhighlight lang="typescript">
	function ~~Mask~~(df, cols, min, max) {		function AddRandomNoise(df, cols, min, max) {
	for (let i = 0; i < CountTop(cols); ++i) {		for (let i = 0; i < CountTop(cols); ++i) {
	let col = cols[i];		let col = cols[i];
Line 184:		Line 184:

	let sourceDatatable = DataTableById(66);		let sourceDatatable = DataTableById(66);
	~~Mask~~(		AddRandomNoise(
	sourceDatatable.SqlDataFrame.Collect(),		sourceDatatable.SqlDataFrame.Collect(),
	["Item_OrderQuantity"],		["Item_OrderQuantity"],
	100,		100,
	50		50
	).Persist(sourceDatatable.Name + "~~_anonymized2~~", #{"ProjectId": sourceDatatable.Project.Id});		).Persist(sourceDatatable.Name + "_randomNoise", #{"ProjectId": sourceDatatable.Project.Id});

	</syntaxhighlight>		</syntaxhighlight>
Line 199:		Line 199:
	sourceDatatable.SqlDataFrame.Collect()		sourceDatatable.SqlDataFrame.Collect()
	.RemoveColumns(["Case Id", "Company Code", "Customer Name"])		.RemoveColumns(["Case Id", "Company Code", "Customer Name"])
	.Persist(sourceDatatable.Name + "~~_anonymized~~", #{"ProjectId": sourceDatatable.Project.Id});		.Persist(sourceDatatable.Name + "_columnsRemoved", #{"ProjectId": sourceDatatable.Project.Id});
	</syntaxhighlight>		</syntaxhighlight>

Ollvihe: /* Generalization: broader categories */

2023-02-05T00:01:16Z

Generalization: broader categories

← Older revision		Revision as of 00:01, 5 February 2023
Line 148:		Line 148:
	"North America": ["USA", "Canada", "Mexico"]		"North America": ["USA", "Canada", "Mexico"]
	}		}
	).Persist(sourceDatatable.Name + "~~_anonymized3~~", #{"ProjectId": sourceDatatable.Project.Id})		).Persist(sourceDatatable.Name + "_anonymized", #{"ProjectId": sourceDatatable.Project.Id})
	</syntaxhighlight>		</syntaxhighlight>

Ollvihe: /* Shuffling */

2023-02-04T23:59:30Z

Shuffling

← Older revision		Revision as of 23:59, 4 February 2023
Line 76:		Line 76:
	Following example anonymizes data by shuffling values in each of the selected columns:		Following example anonymizes data by shuffling values in each of the selected columns:
	<syntaxhighlight lang="typescript">		<syntaxhighlight lang="typescript">
	function ~~Shuffle~~(df, cols) {		function ShuffleData(df, cols) {
	for (let i = 0; i < CountTop(cols); ++i) {		for (let i = 0; i < CountTop(cols); ++i) {
	let shuffledData = Shuffle(NumberRange(0, CountTop(df.Rows) - 1));		let shuffledData = Shuffle(NumberRange(0, CountTop(df.Rows) - 1));
Line 88:		Line 88:

	let sourceDatatable = DataTableById(1);		let sourceDatatable = DataTableById(1);
	~~Shuffle~~(		ShuffleData(
	sourceDatatable.SqlDataFrame.Collect(),		sourceDatatable.SqlDataFrame.Collect(),
	["Case Id", "Company Code", "Customer Name"]		["Case Id", "Company Code", "Customer Name"]

Ollvihe at 23:53, 4 February 2023

2023-02-04T23:53:30Z

← Older revision		Revision as of 23:53, 4 February 2023
Line 1:		Line 1:
	Process mining ~~usually needs to handle confidential~~ data, and thus being able to anonymize data is an essential feature. For example, personally identifiable information (PII) ~~may need to~~ be removed.		Process mining often handles sensitive data, and thus being able to anonymize the data is an essential feature. For example, personally identifiable information (PII) can be removed. The goal of anonymization is hide the sensitive details in the data, while preserving the analytical utility of the data.

	== Principles and best practices ==		== Principles and best practices ==
	Data can be anonymized in ~~the following~~ phases of the data flow:		Data can be anonymized in different phases of the data flow depending on requirements. QPR ProcessAnalyzer provides the following approaches where the data can be anonymized:
	* ~~Within~~ the source system, for example using a view in source database. In this approach, the original data is not fetched to QPR ProcessAnalyzer at all, ~~and~~ the anonymization can be done in a way that even the QPR ProcessAnalyzer administrators don't have access to the original data.		* In the source system, for example using a view in data source system's database. In this approach, the original data is not fetched into QPR ProcessAnalyzer at all, so the anonymization can be done in a way that even the QPR ProcessAnalyzer administrators don't have access to the original data.
	* ~~Within~~ the query to fetch the data from the source system. In this approach, the original data is not fetched to QPR ProcessAnalyzer at all~~, but~~ QPR ProcessAnalyzer developer users may have possibility to see the original data, because they can change the ~~queries to~~ remove the anonymization. Benefit of this approach comparing to the previous is that no changes to the source system is needed ~~for the~~ anonymization.		* In the query to fetch the data from the source system. In this approach like in the previous, the original data is not fetched into QPR ProcessAnalyzer at all. The difference is that QPR ProcessAnalyzer developer users may have possibility to see the original data, because they can change the way the anonymization is done or even remove the anonymization. Benefit of this approach comparing to the previous is that no changes to the source system is needed to implement anonymization.
	* During data ~~is extracted~~ from a source system, ~~it's~~ immediately anonymized in ~~the~~ QPR ProcessAnalyzer ~~side~~ and stored to datatables. In this approach, the original data is not stored permanently to QPR ProcessAnalyzer, and it's not possible to access the original data after the extraction. Similar to the previous approach, developer users have possibility to ~~sett~~ the original data by modifying the script where the anonymization is done.		* During data extraction from the source system, the data is immediately anonymized in QPR ProcessAnalyzer and stored to datatables. In this approach, the original data is not stored permanently to QPR ProcessAnalyzer, and it's not possible to access the original data after the extraction. Similar to the previous approach, developer users have possibility to see the original data by modifying the script where the anonymization is done. In this approach the original data is loaded from the source system into QPR ProcessAnalyzer, but usually this doesn't matter because the transferred data is always encrypted.
	* Original data is loaded into QPR ProcessAnalyzer datatables, data is anonymized and written to other datatables. Benefit of this approach is that the anonymization ~~technique~~ can be changed without reloading the data from the source system. When this approach is used, it's important to set the permissions in a way that users don't have access to the datatables containing the original data.		* Original data is loaded into QPR ProcessAnalyzer datatables, and data is anonymized from those datatables and written to other datatables. Benefit of this approach is that the anonymization method can be changed without reloading the data from the source system. When this approach is used, it's important to set the permissions in a way that users don't have access to the datatables containing the original data.
	* Anonymization is performed when model is loaded into memory. This approach is similar to the previous, except the data is transformed when the model is used (ELT is used instead of ETL). This approach requires to use the loading script.		* Anonymization is performed when model is loaded into memory. This approach is similar to the previous, except the data is transformed when the model is used (ELT is used instead of ETL). This approach requires to use the loading script. The benefit of this approach is that the data doesn't need to be stored twice to datatables. Possible drawback is that the loading might take longer time due to increased computing time to perform the anonymization.
	* Anonymization is performed in dashboard queries for individual dashboards to show the data anonymized. This approach is meant only for testing ~~the~~ anonymization~~, for example~~ to find the suitable technique. This approach ~~should not~~ be used in production, because even though seeing the anonymized data, the dashboard users still have access to the original data in the process mining model.		* Anonymization is performed in dashboard queries for individual dashboards to show the data anonymized. This approach is meant only for testing anonymization methods to find the suitable technique. This approach cannot be used in production, because even though seeing the anonymized data, the dashboard users still have access to the original data in the process mining model.
	* ~~Transformation~~ when exporting data from QPR ProcessAnalyzer: If the original data is stored in ~~the system~~, make sure that users who are only allowed to see the anonymized data, don't have access to the original data.		* If anonymized data is meant to be used outside QPR ProcessAnalyzer, anonymization can be performed when exporting data from QPR ProcessAnalyzer. If the original data is stored in QPR ProcessAnalyzer, make sure that users who are only allowed to see the anonymized data, don't have access to the original data.

	Choosing the most suitable method for anonymization is about ~~finding a balance~~ between the level of privacy and the usefulness of ~~the analysis that can be done from~~ the anonymized data. The more the data is anonymized, the more details are lost which will limit what kind of analysis can be done based on the data.		Choosing the most suitable method for anonymization is all about balancing between the level of privacy and the usefulness of the anonymized data in analytical purposes. The more the data is anonymized, the more details are lost which will limit what kind of analysis and relevant findings can be done from the data. You may get started with the anonymization by gathering the requirements for both
			# What sensitive data should not be revealed
			# What kind of process mining analyses are intended based on the data

	== Anonymization techniques ==		== Anonymization techniques ==