APIM manage workflow with multiple roles APIM 3.0.0 per API based subscription workflow Logging internal HTTP requests Log APIM analytics events to a file Monetization and sample with WSO2 API Manager 2.6.0 Share application and subscription among a set of specific groups or roles WSO2 APIM Correlating analytics event with correlationID APIM analytics distinguish production and sandbox traffic APIM 2.x.x analytics internal and analytics tuneup Configure APIM(Next release) Key Manager User stores APIM(Next release) working with key manager DAS 3.x Parse system variables to Spark Context Revoke OAuth application In APIM 2.1.0 Next WSO2 APIM powered by WSO2 Ballerina Configure WSO2 APIM Analytics on Cluster environment Configure WSO2 DAS 3.1.0 for WSO2 APIM 2.0.0 Analytics WSO2 APIM publishing custom statistics WSO2 APIM Error codes Working with WSO2 message tracer Use DAS admin service to query using Spark SQL Configure WSO2 APIM Analytics using XML WSO2 APIM Generating and Retrieving Custom Statistics Understanding WSO2 APIM Statistics Model Publishing WSO2 APIM 1.10.x Runtime Statistics to DAS with RDBMS Publishing_APIM_1100_Runtime_Statistics_to_DAS Aggregate functions with WSO2 DAS REST API Create a cApp for WSO2 DAS Debugging WSO2 Products using OSGI console. Publishing APIM Runtime Statistics to DAS Deploy cApp on WSO2 DAS How to configure and start the Accumulo minicluster How to setup DNS server on Ubuntu and Ubuntu server How to use Java Reflection how to install apache web server on ubuntu and ubuntu server How to install Mail server on Ubuntu and Ubuntu server How to install squirrelmail webmail client on Ubuntu and Ubuntu Server Pass and return String value to JNI method Pass and return numeric value to JNI method Calling a C Function from the Java Programming Language using JNI AXIS 2 Sample web service Client with maven and eclipse How to setup AXIS 2 with Apache Tomcat AXIS 2 Sample web service with maven and eclipse Robot framework Sample with Selenium Robot framework Custom Library Sample Behaviour-Driven Development with JBehave and Eclipse Play Audio with Netbeans and linking with LibVLC Implement LibVLC based player with QT-part2 Simple Audio playing sample with LibVLC How to install LibVLC on Ubuntu Implement LibVLC based player with QT-part1
APIM 2.x.x analytics internal and analytics tuneup
  1. Introduction

    APIM GWs are publishing analytics event to DAS receiver for each and every request GW serve. Also when DAS receives an event it persists the raw event after do some real-time analysis. And also generate some new event based on the original events and again generate some new tables. These data are processed by batch analytics scripts and summarised data again persist on both DAS and RDBMS.

    As you can see these data are aggregated by DAS and it’s storage getting filled with data over the time. But when the system runs for a long time since these accumulated data still in the system, it affects the performance, storage limitation and unnecessary filtering. This issue is not just happening for DAS, but for RDBMS as well.

    Below is the workaround that can be taken into account overcome this issue. Some of these topics are already found in the APIM. But there is some gap in the system and need to be enhanced. And also when extending the analytics system, these steps need to be taken in order to get better performance for the long-running system.

  2. Incremental analysis

    Incremental analysis is a feature in WSO2 DAS and APIM analytics to optimize the performance in a long-running system which can have big data. A key idea of this feature is it ignores the already processed data next time a spark job run. In Apim, only a few scenarios have used these features. Because not all the events stream received the same amount of traffic. Thus request, response and execution time event stream data are incrementally processed.

    If you have an idea to improve the performance more, it is time to think about the other tables which used spark summarization(Like throttle summary etc) to apply the incremental processing to optimize the performance.

  3. Data Purging

    Data purging is another key factor that affects the performance. But if you enable the incremental processing, it is not affected much to the system even data purging did not enable. But eventually, you need to consider data plugin of the analytics table, since it grows your system space, RDBMS space, and RDBMS performance.

    Currently, APIM doesn't provide data purging configuration for tables except for few alert related table. The reason behind this is, User related data purging is a user’s task and APIM can't decide it. Another thing is you can't just purge tables as you wish because it gonna affect to the other tables and cause loss of data.

    In the APIM analytics table list, there are 3 sets of tables according to my view. Those are,

    • Batch processing related tables

      From that sets, Batch processing related table is the most critical tables in the system. Which data growth is higher than others normally. Also, you just can't purge records in this table directly. As you may know, APIM batch analytics data are summarized daily basis. So less than 24h old data are not yet completely processed and cannot be removed from the tables. Thus we need to keep those data when we apply the purging on these tables set. In the purging configuration, we better mention the time interval of a table to be executed as 2 days which include 1 additional day in order to prevent data loss due to error margin.

      • ORG_WSO2_APIMGT_STATISTICS_WORKFLOW
      • ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST
      • ORG_WSO2_APIMGT_STATISTICS_PERHOURREQUEST
      • ORG_WSO2_APIMGT_STATISTICS_PERDAYREQUEST
      • ORG_WSO2_APIMGT_STATISTICS_PERMINUTERESPONSE
      • ORG_WSO2_APIMGT_STATISTICS_PERHOURRESPONSE
      • ORG_WSO2_APIMGT_STATISTICS_PERDAYRESPONSE
      • ORG_WSO2_APIMGT_STATISTICS_PERMINUTEEXECUTIONTIMES
      • ORG_WSO2_APIMGT_STATISTICS_PERHOUREXECUTIONTIMES
      • ORG_WSO2_APIMGT_STATISTICS_PERDAYEXECUTIONTIMES
      • ORG_WSO2_APIMGT_STATISTICS_THROTTLE
      • ORG_WSO2_APIMGT_STATISTICS_FAULT
      • ORG_WSO2_CARBON_IDENTITY_OAUTH_TOKEN_ISSUANCE
      • LOGANALYZER
    • Alert related tables

      These table set include alert related data.

      In this set, there is two set of tables. Those are streams table and generated alert tables. Streams tables are persisted data of the streams those are generated from the original analytics events. Those data are used to generate alert and then generated alert is saved in the alert tables. Data purging is also can apply on these tables and it does not affect any functionality. But which clean the system alert history. Recommend to keep alert data at least 30 days of history and better to clean the data once it’s expired.

      • streams table
        • ORG_WSO2_ANALYTICS_APIM_APIHEALTHMONITORALERTSTREAM
        • RG_WSO2_ANALYTICS_APIM_RESPONSEPERMINPERAPISTREAM
        • ORG_WSO2_ANALYTICS_APIM_REQUESTPERMINPERAPISTREAM
        • ORG_WSO2_ANALYTICS_APIM_ABNORMALBACKENDTIMEALERTSTREAM
        • ORG_WSO2_ANALYTICS_APIM_ABNORMALREQUESTSPERMINALERTSTREAM
        • ORG_WSO2_ANALYTICS_APIM_ABNORMALRESPONSETIMEALERTSTREAM
        • ORG_WSO2_ANALYTICS_APIM_REQUESTPATTERNCHANGEDSTREAM
        • ORG_WSO2_ANALYTICS_APIM_ALLAPIMALERTSSTREAM
        • ORG_WSO2_ANALYTICS_APIM_REQUESTPERMINSTREAM
      • generated alert tables
        • ORG_WSO2_ANALYTICS_APIM_ALERT_ABNORMALTOKENREFRESH
        • ORG_WSO2_ANALYTICS_APIM_ALERT_UNUSUALIPACCESS
        • ORG_WSO2_ANALYTICS_APIM_REQUESTPERCENTILE
        • ORG_WSO2_ANALYTICS_APIM_RESPONSEPERCENTILE
        • ORG_WSO2_ANALYTICS_APIM_ALERT_ABNORMALTIERUSAGEALERT
        • ORG_WSO2_ANALYTICS_APIM_TIERLIMITHITTINGALERT
        • ORG_WSO2_ANALYTICS_APIM_APIAVAILABILITY
        • ORG_WSO2_ANALYTICS_APIM_ACCESSTOKENREFRESHTIMEDIFFERENCE
        • ORG_WSO2_ANALYTICS_APIM_IPACCESSSUMMARY
    • Configuration related tables

      This table holds the analytics related configuration. So in this table contain the alert subscription-related information. It includes per user type of alert subscribed and email configuration of that subscriber. Hence this table is not allowed to purge. If it purged, all the subscribed users and their subscribed alert may reset. Since this is the only place those config data are available, not will be able to roll it back if it purged.

      • ORG_WSO2_ANALYTICS_APIM_ALERTSTAKEHOLDERINFO
    • Summarization Table

      Following summary tables are used to save summarized or processed data in the DAS before it saved on external RDBMS. But these tables are deprecated from AM 2.1.0. So after the AM 2.1.0 these tables may not created or populated.

      If this tables are available in your system, you can purge them as well. This is the original copy of the RDBMS summary tables and after the data are moved to RDBMS purging can be initiated. Like the raw event table, condition for this table is also purge 2 days old data. Because these tables are update every time the batch analytics job is execute within the day. Even this summary tables are purged from the system, it not affect to the RDBMS summary tables and it’s data may not disappear or not affect to the previous summary data.

      • API_EXE_TME_DAY_SUMMARY
      • API_LAST_ACCESS_TIME_SUMMARY
      • API_REQUEST_SUMMARY
      • API_REQUEST_USER_AGENT_SUMMARY
      • API_THROTTLED_OUT_SUMMARY
      • API_REQ_USER_BROW_SUMMARY
      • API_EXE_TIME_HOUR_SUMMARY
      • API_VERSION_USAGE_SUMMARY
      • API_RESOURCE_USAGE_SUMMARY
      • API_RESPONSE_SUMMARY
      • API_EXE_TIME_MIN_SUMMARY

      These tables are also summarized data of the logs analytics events. These tables are populated only if log analytics is enabled. These data are only stored in the Analyzer tables and APIM dashboard fetch data over Analytics REST API. These tables are allowed to purge as per the previous summary tables

      • LOGANALYZER_CLASS_LEVEL_ERROR_MONTHLY
      • LOGANALYZER_CLASS_LEVEL_ERROR_WEEKLY
      • LOGANALYZER_CLASS_LEVEL_ERROR_DAILY
      • LOGANALYZER_MESSAGE_LEVEL_ERROR_MONTHLY
      • LOGANALYZER_MESSAGE_LEVEL_ERROR_WEEKLY
      • LOGANALYZER_MESSAGE_LEVEL_ERROR_DAILY
      • LOGANALYZER_APIM_ARTIFACT_DEPLOYED_DAILY
      • LOGANALYZER_APIM_ARTIFACT_DELETED_DAILY
      • LOGANALYZER_APIM_MESSAGE_PROCESSING_MONTHLY
      • LOGANALYZER_APIM_MESSAGE_PROCESSING_WEEKLY
      • LOGANALYZER_APIM_MESSAGE_PROCESSING_DAILY
      • LOGANALYZER_INVALID_LOGIN_ATTEMPT_MONTHLY
      • LOGANALYZER_INVALID_LOGIN_ATTEMPT_WEEKLY
      • LOGANALYZER_INVALID_LOGIN_ATTEMPT_DAILY
      • LOGANALYZER_APIM_AUDIT_LOG
      • LOGANALYZER_APIKEY_STATUS
  4. Data Archiving

    Data archiving can be performed on both Analytics server databases and APIM statistics database. Objective is to reduce the number of records in the database in order to optimize the system by freeing up disk space and reduce time taken to execute sql queries on data base on both analytics and apim servers.

    Analytics server data can be remove from the system by enabling data purging configuration using Analytics server purging feature. But data archiving is required if the data need to be backup for further analysis and if there is requirement to restored.

    In the statistics data which is summary of the original analytics events can also to be archived. Archiving on summary data may be required if system need to be tune up. Also raw data are expected to be purged, summary data should be available for further analysis. Another use case for data archiving from the statistics database is, sometime summary data are process further for higher time span(ex: per day records to per year summary) and store in the same DB or somewhere else. So the processed data or per day records can be clean from the system, since new representation of the same data is present.

    So this is optional task in order to performance improvement or system storage clean. Way to perform is depend on the underlying database.

Add Comment

* Required information
1000
Powered by Commentics

Comments (0)

No comments yet. Be the first!