QPR ProcessAnalyzer Logs: Difference between revisions

From QPR ProcessAnalyzer Wiki
Jump to navigation Jump to search
(TK-53330)
 
(62 intermediate revisions by one other user not shown)
Line 1: Line 1:
QPR ProcessAnalyzer logs operations to several kinds of logs that can be used for auditing, troubleshooting or other purposes. QPR ProcessAnalyzer collects the following logs:
QPR ProcessAnalyzer writes a permanently stored log of all operations it's performing. The logs can be used for monitoring, auditing, troubleshooting and other purposes. The main types of logs are [[#Task Log|task log]], [[#Script Log|script log]] and [[#Server File Log|server file log]]. The task log (and visual reports based on it) can be used to monitor the system usage and collect statistics. The script log are collected for each script when the script runs. The server file log is not needed in every-day operations, but if there are exceptional situations, such as system malfunctioning, the server file log is helpful in investigating what has happened.
* [[Operation Log]]: logs all the operations performed in QPR ProcessAnalyzer.
* [[Excel Log]]: logs operations on the computer where the QPR ProcessAnalyzer Excel client is installed.
* [[#Progress Log|Progress Log]]: logs recent operations in QPR ProcessAnalyzer with status messages.
* [[#Script Log|Script Log]]: logs information about scripts that are run.
* [[Installing_QPR_ProcessAnalyzer_Excel_Client#Enabling Installation Debug Logs for QPR ProcessAnalyzer|Installation Log]]: logs information about installation.
* [[QPR ProcessAnalyzer Excel Client Progress Status|Log in Progress Status dialog]]: shows logs of an ongoing operation.


==Progress Log==
== Task Log ==
The Progress Log contains recent operations done in QPR ProcessAnalyzer, showing date, time, type, name and id of the operation, as well as the performance information about the operation. Also, at the beginning of each line, the source of the information is listed. These can be one of the following:
All user created requests to the [[QPR_ProcessAnalyzer_API|Web API]] are recorded as tasks, such as login, logout, modifying data, running queries etc. The Task Log can be browsed in the UI by opening the '''System Reports''' in the navigation menu (available only for system administrators). There are following task reports available:
* '''Server''': shows operation status messages stored in the server side
* '''Recent tasks''': List of recent tasks in the order of start time.
* '''Client''': shows operation status messages stored in the client side
* '''Running tasks''': All tasks currently being executed.
* '''cLogs''': shows Client.log messages from the client (the same entries as in [[Excel Log]])
* '''Tasks by time''': Trend over time how task counts have changed.
* '''Task duration''': Distribution of task counts of different durations.
* '''Most used models''': List of models that are have been queried most.
* '''Most active users''': List of users who have used the system most actively.
* '''User logins by time''': Table of how many times users have logged in to the system over time.
* '''Running tasks at specific time''': This report can be checked afterwards what kind of activity was in progress the system at a specific time. It can be used to investigate for example experienced performance issues.


The start of each new operation is marked in the following way:
Currently running tasks can be stopped by the administrator both in the ''Recent tasks'' and ''Running tasks'' reports by selecting the task(s) and in the right click context menu selecting '''End tasks'''.
<pre>
------------------ NEW OPERATION START -----------------
</pre>


You can copy and paste the contents of the log for troubleshooting purposes.  
Note that loading of a model is visible as a separate task in the task log, besides to the task that initiated the model loading.


A maximum of 100,000 rows is shown in the log. If there is more data than that, the log cuts it from the beginning, so the latest information is always shown. The log size (in characters) is shown at the end of the log.
Task log have for example the following use cases:
* To see the most active and idle times of the system to plan the optimal time for system updates and maintenance.
* In case of performance issues, how much load the is in the system in different times.
* How active the usage of the system is and how many users are using the system.
* Which content and models are browsed by the users in the system.


==Script Log==
=== Task Log Settings ===
The Script Log contains information about the script that is currently running and scripts that have been run. The log contains, for example, the output of the WriteLog and Print commands used in the script, as well as details about possible errors. After a script execution has finished, you can access the Script Log by clicking the '''Show Details''' button.
The following settings are available for the task log (in the '''Analyze''' tab):
* '''Time from''': Show only tasks that have been started later than this time.
* '''Time to''': Show only tasks that have been started earlier than this time.
* '''Only running tasks''': Show only tasks that are currently running.
* '''Only cancelled tasks''': Show only tasks that have been requested to cancel.
* '''Only model related tasks''': Show only tasks that are related to a model.
* '''Only user related tasks''': Show only tasks that are related to a user.
* '''Only logins''': Show only login activities.
* '''Only tasks longer than''': Show only tasks where duration is longer is specified.


[[File:Script_log.png]]
* '''Maximum Text Length''': Defines the maximum number of characters shown in the 'Additional Data' column.
* '''Only logins''': Show only log entries of successful authentications by users.
* '''Maximum rows''': Maximum number of rows shown by the log (default max. 1000).


==QPR ProcessAnalyzer Excel Client Progress Status Log==
The following columns can be selected for the task log (in the '''Columns''' tab):
When you run an operation in QPR ProcessAnalyzer Excel Client that takes longer than two seconds, the '''Progress Status''' dialog shows its progress while the operation is running. This dialog shows the operation name and id (in parentheses), the progress of the operation with a progress bar indicator and the elapsed time. Also, when you extend the dialog window by dragging from the bottom right corner, Excel activity is shown under the progress bar indicator as follows:
* '''Task type''': Type of the task, e.g. run expression, run script, etc.
* '''CPU''': shows the percentage of CPU utilization (per core)
* '''Start Time''': Start time of the task.
* '''Memory''': shows the memory usage in megabytes
* '''End Time''': End time of the task. This field is empty if the task is running.
* '''In''': shows the approximate amount of megabytes currently being received
* '''Task duration''': Duration of the task. Shown only if the task is completed.
* '''Out''': shows the approximate amount of megabytes currently being sent
* '''Task cancelled''': Whether the task was requested to be cancelled.
* '''Server logs updated''': shows the timestamp when the [[Progress Log]] was last updated
* '''Task parameters''': Input parameters of the task.
* '''Task result''': Exceptional results of the task, e.g., an error message.
* '''Model name''': Name of model for which the task was performed.
* '''Model Id''': Id of model for which the task was performed.
* '''User name''': Login name of the user performed the task.
* '''User full name''': Full name of the user performed the task.
* '''User Id''': Id of user performed the task.
* '''Session Id''': User session id that performed the task.
* '''Task Id''': Unique id of the task.


When the dialog is resized, there is also a list of status messages including the timestamp of each message. For more information on these status messages, see [[Progress Log]].
== Script Log ==
The script log is collected for each run of a script. The script log contains possible errors in the script run and also log writes that are explicitly added to the code of the script. See more in [[Managing_Scripts#Viewing_Script_Log|Managing Scripts]] how to use the script log. Note that in the UI, you can only see the last completed script run, although the system is storing the entire run history.


[[File:progress_status.png|Progress Status dialog]]
Script logs contain following information:
* All errors occurring during the script run (script run also stops when an error occurs).
* All log writes made by commands in the script code (in expression scripts ''WriteLog'' function calls, and in SQL scripts ''--#WriteLog'' and ''PRINT'' commands).
* For SQL scripts, all executed scripting commands (commands starting with ''--#'').


While the operation is running, you can browse the contents of the dialog and also copy and paste text from it for troubleshooting purposes, for example. To terminate the operation before it is completed, click '''Cancel'''. The dialog will close automatically once the operation has completed.
== Server File Log ==
The server file log contains information about all operations performed by the QPR ProcessAnalyzer server in a detailed level. The server file log is stored in the location defined by the [[Installing_QPR_ProcessAnalyzer_Server#Server_settings_file_(appsettings.json)|LogFilePath setting in the appsettings.json file]]. The Web.config file also contains [[Installing_QPR_ProcessAnalyzer_Server#Server_settings_file_(appsettings.json)|several other settings]] that affect the logging behavior, e.g. maximum log file size and file rotation.
 
All log entries are classified to the following logging levels based on their importance for the system administrator:
* '''FATAL''': (currently not in use) For failures that are likely application wide, and the entire application may have stopped working. Thus it requires immediate attention from the system administrator.
* '''ERROR''': For failures that cause handling of the current request to fail, but the failure is not application wide, and the application should continue responding to other requests. System administrator should soon investigate the reason for these log entries. If the failed requests serve important functionality in the system, these failures might be critical to users.
* '''WARN''': Abnormal or unexpected event has occurred in the application, but handling of the current request did not fail. No immediate resolution is required, but the system administrator should investigate the reason for these log entries when suitable.
* '''INFO''': Tracks the successful progress of the request handling or other flow in the application. Normally these log entries can be ignored by the system administrator, unless an unknown failure has occurred and more detailed information is needed.
* '''DEBUG''': These are technical details for application developers, so these log entries don't have any use for system administrators.
 
The following log entries for security auditing purposes are made to the server file log (using ''Information'' log level):
* When specifying a non-existing username, the following log entry is made: ''User is not found: <username>''
* When specifying wrong password for an existing user account, the following log entry is made: ''User password is invalid: <username>''
* When trying to log in with an inactivated user account, the following log entry is made: ''User account is invalid: <username>''
 
== Dashboard Usage ==
The dashboard usage reports can be browsed in the UI by opening the '''System Reports''' in the navigation menu (available only for system administrators). There are following dashboard usage reports available:
* '''Popular dashboards''': Shows dashboards usage in a table of maximum of 500 dashboards which have the largest number of opens during the defined period.
* '''Usage by time''': Shows dashboards usage in a pivot table where dashboards are on rows and time periods on columns.
* '''Usage by users''': Shows dashboards usage in a pivot table where dashboards are on rows and users on columns.
* '''Specific dashboards''': Shows the selected dashboards usage in a pivot table where users who opened the dashboards are on rows and time periods on columns.

Latest revision as of 11:41, 4 June 2024

QPR ProcessAnalyzer writes a permanently stored log of all operations it's performing. The logs can be used for monitoring, auditing, troubleshooting and other purposes. The main types of logs are task log, script log and server file log. The task log (and visual reports based on it) can be used to monitor the system usage and collect statistics. The script log are collected for each script when the script runs. The server file log is not needed in every-day operations, but if there are exceptional situations, such as system malfunctioning, the server file log is helpful in investigating what has happened.

Task Log

All user created requests to the Web API are recorded as tasks, such as login, logout, modifying data, running queries etc. The Task Log can be browsed in the UI by opening the System Reports in the navigation menu (available only for system administrators). There are following task reports available:

  • Recent tasks: List of recent tasks in the order of start time.
  • Running tasks: All tasks currently being executed.
  • Tasks by time: Trend over time how task counts have changed.
  • Task duration: Distribution of task counts of different durations.
  • Most used models: List of models that are have been queried most.
  • Most active users: List of users who have used the system most actively.
  • User logins by time: Table of how many times users have logged in to the system over time.
  • Running tasks at specific time: This report can be checked afterwards what kind of activity was in progress the system at a specific time. It can be used to investigate for example experienced performance issues.

Currently running tasks can be stopped by the administrator both in the Recent tasks and Running tasks reports by selecting the task(s) and in the right click context menu selecting End tasks.

Note that loading of a model is visible as a separate task in the task log, besides to the task that initiated the model loading.

Task log have for example the following use cases:

  • To see the most active and idle times of the system to plan the optimal time for system updates and maintenance.
  • In case of performance issues, how much load the is in the system in different times.
  • How active the usage of the system is and how many users are using the system.
  • Which content and models are browsed by the users in the system.

Task Log Settings

The following settings are available for the task log (in the Analyze tab):

  • Time from: Show only tasks that have been started later than this time.
  • Time to: Show only tasks that have been started earlier than this time.
  • Only running tasks: Show only tasks that are currently running.
  • Only cancelled tasks: Show only tasks that have been requested to cancel.
  • Only model related tasks: Show only tasks that are related to a model.
  • Only user related tasks: Show only tasks that are related to a user.
  • Only logins: Show only login activities.
  • Only tasks longer than: Show only tasks where duration is longer is specified.
  • Maximum Text Length: Defines the maximum number of characters shown in the 'Additional Data' column.
  • Only logins: Show only log entries of successful authentications by users.
  • Maximum rows: Maximum number of rows shown by the log (default max. 1000).

The following columns can be selected for the task log (in the Columns tab):

  • Task type: Type of the task, e.g. run expression, run script, etc.
  • Start Time: Start time of the task.
  • End Time: End time of the task. This field is empty if the task is running.
  • Task duration: Duration of the task. Shown only if the task is completed.
  • Task cancelled: Whether the task was requested to be cancelled.
  • Task parameters: Input parameters of the task.
  • Task result: Exceptional results of the task, e.g., an error message.
  • Model name: Name of model for which the task was performed.
  • Model Id: Id of model for which the task was performed.
  • User name: Login name of the user performed the task.
  • User full name: Full name of the user performed the task.
  • User Id: Id of user performed the task.
  • Session Id: User session id that performed the task.
  • Task Id: Unique id of the task.

Script Log

The script log is collected for each run of a script. The script log contains possible errors in the script run and also log writes that are explicitly added to the code of the script. See more in Managing Scripts how to use the script log. Note that in the UI, you can only see the last completed script run, although the system is storing the entire run history.

Script logs contain following information:

  • All errors occurring during the script run (script run also stops when an error occurs).
  • All log writes made by commands in the script code (in expression scripts WriteLog function calls, and in SQL scripts --#WriteLog and PRINT commands).
  • For SQL scripts, all executed scripting commands (commands starting with --#).

Server File Log

The server file log contains information about all operations performed by the QPR ProcessAnalyzer server in a detailed level. The server file log is stored in the location defined by the LogFilePath setting in the appsettings.json file. The Web.config file also contains several other settings that affect the logging behavior, e.g. maximum log file size and file rotation.

All log entries are classified to the following logging levels based on their importance for the system administrator:

  • FATAL: (currently not in use) For failures that are likely application wide, and the entire application may have stopped working. Thus it requires immediate attention from the system administrator.
  • ERROR: For failures that cause handling of the current request to fail, but the failure is not application wide, and the application should continue responding to other requests. System administrator should soon investigate the reason for these log entries. If the failed requests serve important functionality in the system, these failures might be critical to users.
  • WARN: Abnormal or unexpected event has occurred in the application, but handling of the current request did not fail. No immediate resolution is required, but the system administrator should investigate the reason for these log entries when suitable.
  • INFO: Tracks the successful progress of the request handling or other flow in the application. Normally these log entries can be ignored by the system administrator, unless an unknown failure has occurred and more detailed information is needed.
  • DEBUG: These are technical details for application developers, so these log entries don't have any use for system administrators.

The following log entries for security auditing purposes are made to the server file log (using Information log level):

  • When specifying a non-existing username, the following log entry is made: User is not found: <username>
  • When specifying wrong password for an existing user account, the following log entry is made: User password is invalid: <username>
  • When trying to log in with an inactivated user account, the following log entry is made: User account is invalid: <username>

Dashboard Usage

The dashboard usage reports can be browsed in the UI by opening the System Reports in the navigation menu (available only for system administrators). There are following dashboard usage reports available:

  • Popular dashboards: Shows dashboards usage in a table of maximum of 500 dashboards which have the largest number of opens during the defined period.
  • Usage by time: Shows dashboards usage in a pivot table where dashboards are on rows and time periods on columns.
  • Usage by users: Shows dashboards usage in a pivot table where dashboards are on rows and users on columns.
  • Specific dashboards: Shows the selected dashboards usage in a pivot table where users who opened the dashboards are on rows and time periods on columns.