APIM manage workflow with multiple roles APIM 3.0.0 per API based subscription workflow Logging internal HTTP requests Log APIM analytics events to a file Monetization and sample with WSO2 API Manager 2.6.0 Share application and subscription among a set of specific groups or roles WSO2 APIM Correlating analytics event with correlationID APIM analytics distinguish production and sandbox traffic APIM 2.x.x analytics internal and analytics tuneup Configure APIM(Next release) Key Manager User stores APIM(Next release) working with key manager DAS 3.x Parse system variables to Spark Context Revoke OAuth application In APIM 2.1.0 Next WSO2 APIM powered by WSO2 Ballerina Configure WSO2 APIM Analytics on Cluster environment Configure WSO2 DAS 3.1.0 for WSO2 APIM 2.0.0 Analytics WSO2 APIM publishing custom statistics WSO2 APIM Error codes Working with WSO2 message tracer Use DAS admin service to query using Spark SQL Configure WSO2 APIM Analytics using XML WSO2 APIM Generating and Retrieving Custom Statistics Understanding WSO2 APIM Statistics Model Publishing WSO2 APIM 1.10.x Runtime Statistics to DAS with RDBMS Publishing_APIM_1100_Runtime_Statistics_to_DAS Aggregate functions with WSO2 DAS REST API Create a cApp for WSO2 DAS Debugging WSO2 Products using OSGI console. Publishing APIM Runtime Statistics to DAS Deploy cApp on WSO2 DAS How to configure and start the Accumulo minicluster How to setup DNS server on Ubuntu and Ubuntu server How to use Java Reflection how to install apache web server on ubuntu and ubuntu server How to install Mail server on Ubuntu and Ubuntu server How to install squirrelmail webmail client on Ubuntu and Ubuntu Server Pass and return String value to JNI method Pass and return numeric value to JNI method Calling a C Function from the Java Programming Language using JNI AXIS 2 Sample web service Client with maven and eclipse How to setup AXIS 2 with Apache Tomcat AXIS 2 Sample web service with maven and eclipse Robot framework Sample with Selenium Robot framework Custom Library Sample Behaviour-Driven Development with JBehave and Eclipse Play Audio with Netbeans and linking with LibVLC Implement LibVLC based player with QT-part2 Simple Audio playing sample with LibVLC How to install LibVLC on Ubuntu Implement LibVLC based player with QT-part1
Understanding WSO2 APIM Statistics Model
  1. Introduction

    In this Document describe and illustrate the APIM Statistic publishing and generating model. In this Document, it describes the internal component of the APIM, External Analyzer information and other data retrieval components.

    APIM generate an event based on the APIM invocation pattern and published to all the listening event analyzers. The analyzer is responsible to accumulate all the event and generate Summary data based on the defined summarization logic(cApp). After the summarized data are generated APIM Dashboard can retrieve statistics from the Analyzer data-source to UI via APIM analytics client.

    a more descriptive component is illustrated in the below diagram.

  2. APIM Usage Publisher

    Internal APIM component listening to the APIM invocations and it’s behavior. Based on the request and responses it generated event and published to all the event receivers. This publisher publish following event streams,

    • org.wso2.apimgt.statistics.request

    • org.wso2.apimgt.statistics.response

    • org.wso2.apimgt.statistics.fault

    • org.wso2.apimgt.statistics.throttle

  3. BAM Mediator

    this will publish event stream org_wso2_apimgt_statistics_destination.

  4. Workflow Executor

    APIM component will publish event stream org.wso2.apimgt.statistics.workflow containing work-flow related event data.

  5. APIM Usage Client

    The client which is responsible to retrieve statistics data from the available data source. There can be more than one usage client in APIM. In default, there is 2 usage client and either one can use at a time. APIM 1.10.x is composed of two usage client REST and RDBMS client. A user can configure to use either client. By default, REST client is configured and it uses DAS rest API to fetch summarized data.

    In the diagram, you can find the path having red and black color. In black color, it demonstrates the path when APIM configure with REST client. In red color, it demonstrates the RDBMS client data flow path. In that case, it stores the same identical data store in the DAS internal table into external RDBMS. These two clients use different summary data generating plan defined in the API_Manager_Analytics_RDBMS.car and API_Manager_Analytics_REST.car.

  6. Data Analyzer

    Summary data generator based on the received event. APIM 1.10.x use DAS 3.0.x to perform analytics task. DAS use apache spark as a processing language. APIM have a pluggable component named DAS Capp which can be deployed into DAS In that cApp contain the all the logic to summarized data and it’s the destination. APIM has two inbuilt capp API_Manager_Analytics_RDBMS.car and API_Manager_Analytics_REST.car. Both contain the same summarization logic except rest client final destination is DAS internal database and an RDBMS it is external RDBMS.

  7. APIM event Streams

    APIM 1.10.0 provide 6 types of the event streams as below.

    1. org.wso2.apimgt.statistics.request

      this stream track the information for the API request.

      consumerKey     :       STRING
      context         :       STRING
      api_version     :       STRING
      api             :       STRING
      resourcePath    :       STRING
      method          :       STRING
      version         :       STRING
      request         :       INT
      requestTime     :       LONG
      userId          :       STRING
      tenantDomain    :       STRING
      host            :       STRING
      apiPublisher    :       STRING
      application     :       STRING
      applicationId   :       STRING
      userAgent       :       STRING
      tier            :       STRING
      throttledOut    :       BOOL
      
    2. org.wso2.apimgt.statistics.response

      this stream track the information for the API response. It includes the time taken to response get back and received time etc.

      consumerKey     :       STRING
      context         :       STRING
      api_version     :       STRING
      api             :       STRING
      resourcePath    :       STRING
      method          :       STRING
      version         :       STRING
      response        :       INT
      responseTime    :       LONG
      serviceTime     :       LONG
      backendTime     :       LONG
      user            :       STRING
      eventTime       :       LONG
      tenantDomain    :       STRING
      host            :       STRING
      apiPublisher    :       STRING
      application     :       STRING
      applicationId   :       STRING
      cacheHit        :       BOOL
      responseSize    :       LONG
      protocol        :       STRING
        
    3. org.wso2.apimgt.statistics.fault

      contain the fault API invocations. It includes the API with backend error, timeout etc

      consumerKey     :       STRING
      context         :       STRING
      api_version     :       STRING
      api             :       STRING
      resourcePath    :       STRING
      method          :       STRING
      version         :       STRING
      errorCode       :       STRING
      errorMessage    :       STRING
      requestTime     :       LONG
      userId          :       STRING
      tenantDomain    :       STRING
      host            :       STRING
      apiPublisher    :       STRING
      application     :       STRING
      applicationId   :       STRING
      protocol        :       STRING
        
    4. org_wso2_apimgt_statistics_destination

      this related information based on the endpoint URL

      tenant_id           :       INT
      http_method         :       STRING
      character_set_encoding:       STRING
      remote_address      :       STRING
      transport_in_url    :       STRING
      message_            :       STRING
      remote_host         :       STRING
      service_prefix      :       STRING
      host                :       STRING
      message_direction   :       STRING
      operation_          :       STRING
      message_id          :       STRING
      timestamp           :       LONG
      api                 :       STRING
      version             :       STRING
      request             :       INT
      apiPublisher        :       STRING
      context             :       STRING
      destination         :       STRING
      requestTime         :       LONG
      host                :       STRING
        
    5. org.wso2.apimgt.statistics.throttle

      it contains the API invocation with throttle info, throttling can happen due to the following reason.

      • application limit exceed
      • resource limit exceed
      • API limit exceeded
      • hard level limit exceed

      information based on the above throttle is conform with above event

      accessToken         :       STRING
      userId              :       STRING
      tenantDomain        :       STRING
      api                 :       STRING
      api_version         :       STRING
      context             :       STRING
      apiPublisher        :       STRING
      throttledTime       :       LONG
      application         :       STRING
      applicationId       :       STRING
      throttledOutReason  :       STRING
        
    6. org.wso2.apimgt.statistics.workflow

      create an event based on the APIM work-flow and publish to the analyzer

      workflowReference   :       STRING
      workflowStatus      :       STRING
      tenantDomain        :       STRING
      workflow            :       STRING
      createdTime         :       LONG
      updatedTime         :       LONG
        
  8. APIM Summarized tables (In DAS Internals)

    Note that, These Summarized tables are stored in DAS internal storage. Both two cApps APIM deploy on DAS, will first store summary data into these Tables. There is an additional column in some of these tables which contain the composition column of other columns. Those columns type are facet type and which is used to support aggregation function on DAS REST API. Also, all the columns are indexed in order to search using Lucene and support DAS REST API. When APIM configured with RDBMS client all of these tables are replicated in external RDBMS removing facet columns.

    • API_REQUEST_SUMMARY

      contain the summary data of the request event stream.

      API_REQUEST_SUMMARY table schema

      CREATE TEMPORARY TABLE API_REQUEST_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_REQUEST_SUMMARY",
        schema "api string -i,
        api_version string -i,
        version string -i,
        apiPublisher string -i,
        consumerKey string -i,
        userId string -i,
        context string -i,
        max_request_time long -i,
        total_request_count int -i,
        hostName string -i,
        year int -i,
        month int -i,
        day int -i,
        time string -i,
        key_api_facet facet -i,
        key_userId_facet facet -i,
        api_version_userId_facet facet -i,
        api_version_userId_apiPublisher_facet facet -i,
        api_version_userId_context_facet facet -i",
        primaryKeys "api,api_version,version,apiPublisher,consumerKey,userId,context,hostName,year,month,day"
        );
      
    • API_VERSION_USAGE_SUMMARY

      contain the summary data for APIM usage vise usage. it is also derived from the request event table

      API_VERSION_USAGE_SUMMARY table schema

      CREATE TEMPORARY TABLE API_VERSION_USAGE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_VERSION_USAGE_SUMMARY",
        schema "api string -i,
        version string -i,
        apiPublisher string -i,
        context string -i,
        total_request_count int -i,
        hostName string -i,
        year int -i,
        month int -i,
        day int -i,
        time string -i,
        max_request_time long -i,
        api_version_context_facet facet -i",
        primaryKeys "api,version,apiPublisher,context,hostName,year,month,day"
        );
      
    • API_Resource_USAGE_SUMMARY

      contain the summarized data for APIM usage by resources and it is also derived from the request event table

      API_Resource_USAGE_SUMMARY table schema

        CREATE TEMPORARY TABLE API_Resource_USAGE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_Resource_USAGE_SUMMARY",
        schema "api string -i,
        version string -i,
        apiPublisher string -i,
        consumerKey string -i,
        resourcePath string -i,
        context string -i,
        method string -i,
        total_request_count int -i,
        hostName string -i,
        year int -i,
        month int -i,
        day int -i,
        time string -i,
        max_request_time long -i,
        key_api_method_path_facet facet -i,
        api_version_context_method_facet facet -i",
        primaryKeys "api,version,apiPublisher,consumerKey,context,resourcePath,method,hostName,year,month,day"
        );
      
    • API_RESPONSE_SUMMARY

      contain the summarized data from API responses. it is derived from the response event table

      API_RESPONSE_SUMMARY table schema

        CREATE TEMPORARY TABLE API_RESPONSE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_RESPONSE_SUMMARY",
        schema "api_version string -i,
        apiPublisher string -i,
        context string -i,
        serviceTime int -i,
        total_response_count int -i,
        hostName string -i,
        year int -i,
        month int -i,
        day int -i,
        time string -i,
        max_request_time long -i,
        api_version_context_facet facet -i",
        primaryKeys "api,version,apiPublisher,context,hostName,year,month,day"
        );
      
    • API_FAULT_SUMMARY

      contain the summarized data of faulty API invocation. it is derived from the fault event stream

      API_FAULT_SUMMARY table schema

        CREATE TEMPORARY TABLE API_FAULT_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_FAULT_SUMMARY",
        schema "api string -i,
        version string -i,
        apiPublisher string -i,
        consumerKey string -i,
        context string -i,
        total_fault_count int -i,
        hostName string -i,
        year int -i,
        month int -i,
        day int -i,
        time string -i,
        max_request_time long -i,
        consumerKey_api_facet facet -i,
        api_version_apiPublisher_context_facet facet -i",
        primaryKeys "api,version,apiPublisher,consumerKey,context,hostName,year,month,day"
        );
      
    • API_DESTINATION_SUMMARY

      contain the summarized data of API destination. and it is derived from the destination event stream

      API_DESTINATION_SUMMARY table schema

        CREATE TEMPORARY TABLE API_DESTINATION_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_DESTINATION_SUMMARY",
        schema "api string -i,
        version string -i,
        apiPublisher string -i,
        context string -i,
        destination string -i,
        total_request_count int -i,
        hostName string -i,
        year int -i,
        month int -i,
        day int -i,
        time string -i,
        max_request_time long -i,
        api_version_context_dest_facet facet -i",
        primaryKeys "api,version,apiPublisher,context,destination,hostName,year,month,day"
        );
      
    • API_LAST_ACCESS_TIME_SUMMARY

      contain the summary data of the last access time of the API, it is also derived from the request event stream

      API_LAST_ACCESS_TIME_SUMMARY table schema

        CREATE TEMPORARY TABLE API_LAST_ACCESS_TIME_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_LAST_ACCESS_TIME_SUMMARY",
        schema "tenantDomain string -i,
        apiPublisher string -i,
        api string -i,
        version string -i,
        userId string -i,
        context string -i,
        max_request_time long -i",
        primaryKeys "tenantDomain,apiPublisher,api"
        );
      
    • API_THROTTLED_OUT_SUMMARY

      contain the summary of the throttle out API invocation data. it is derived from the throttle out event table and request table

      API_THROTTLED_OUT_SUMMARY table schema

      CREATE TEMPORARY TABLE THROTTLED_OUT_FINAL_SUMMARY USING CarbonAnalytics OPTIONS (tableName "API_THROTTLED_OUT_SUMMARY",
        schema "api string -i,
        api_version string -i,
        context string -i,
        apiPublisher string -i,
        applicationName string -i,
        tenantDomain string -i,
        year int -i,
        month int -i,
        day int -i,
        week int -i,
        time string -i,
        success_request_count int -i,
        throttleout_count int -i,
        max_request_time long -i,
        api_year_month_week_day_facet facet -i,
        applicationName_facet facet -i",
        primaryKeys "api,api_version,context,apiPublisher,applicationName,tenantDomain,year,month,day"
      );
      
  9. APIM Available Statistics
    • Publisher

      In Publisher there are two categories All statistic and statistic section. both contain the following types of statistics. In all statistics section, you can see the statistics for the current tenant domain. For the statistics section, you can see the data for the current logging publisher related information.

      • API Subscriptions

        list of APIs with its current subscriptions

      • API Usage

        List of APIs with it’s hit count

      • API Response Times

        List APIs with its average response time

      • API Last Access Times

        List of APIs with it’s last access time

      • API Usage by Resource Path

        List of APIs with it’s hit count for each resource and methods

      • API Usage by Destination

        API endpoint hit counts

      • API Usage Comparison

        API hits counts and hit count comparison with all other APIs

      • API Throttled Out Requests

        API success and throttle out count by APIs and application

      • App Throttled Out Requests

        API success and throttle out count by application

      • Faulty Invocations

        API fault count and its percentage with respect to all request count

    • Store
      • API Usage per Application

        API usage by application wise

      • Top Users per Application

        top users based on the application

      • API Usage from Resource Path per Application

        API resources for API and application

      • Faulty Invocations per Application

        Faulty API invocation by application wise

Add Comment

* Required information
1000
Powered by Commentics

Comments (0)

No comments yet. Be the first!