Hybrid Multi-Cloud Monitoring through AWS Managed Services for Grafana
Customers with hybrid multi-cloud infrastructures often times have to deal with multi tools and technologies into their Operations observability management. Where the customers have difficulties in achieving the visibility and control to their hybrid cloud environments.
Amazon Managed Services for Grafana (AMG) is a fully managed and secure data visualization service that enables customers to complete their own observability picture, reaching across logs, metrics, traces and other applications with enterprise data sources available out of the box. AMG leverages the open source Grafana project, a widely deployed data visualization tool popular for its extensible data source support. Jointly developed with Grafana Labs, AMG helps our customers to achieve success with their operations observability strategy. AMG is a fully managed services from AWS which manages the service provisioning, setup, upgrades, scaling and maintenance of Grafana by eliminating the need for customers to have additional management overhead by themselves.
AMG supports many different out of the box data sources, which are storage backends that you can query in Grafana, such as to build dashboards. Each data source has a specific query editor that is customized for the features and capabilities that the particular data source exposes.
AMG along with AWS monitoring data sources such as Amazon CloudWatch, Amazon Elasticsearch Services, AWS IOT SiteWise, Amazon Managed Service for Prometheus (AMP), Amazon Timestream, and AWS X-Ray also provides data sources for hybrid multi-clouds such as Azure, GCP that will help you to build dashboards for hybrid Multi-Cloud Monitoring metrics in a single pane of glass.
This blog details how AWS Managed Services for Grafana enhances the enterprise capabilities in managing your operations observability by centralizing the hybrid multi-cloud environment single pane of glass. AMG natively integrates with AWS services and other public cloud providers data sources so you can securely add, query, visualize, and analyze your AWS data across multiple accounts and Regions with a few clicks in the AWS Console.
Solution Overview
AWS Managed Services for Grafana is a powerful visualization and alerting tool with access to enterprise data sources plugins that provides better scalability, collaboration and governance for operations management in a self-managed environment.
The Architecture diagram above consists of simulated Hybrid multi-cloud deployments on various public cloud providers and on-premisses that will consume the data sources in AMG as one of the data ingestion patterns for implementing the best practices.
Every customer has a unique application portfolio workloads that place requirements and constraints on the architecture of hybrid or multi-cloud setup. The pattern listed in this solution overview will cover the following aspect:
· A Single pane of hybrid multi-cloud deployment using AMG and centralized logging
This blog details overview on 4 steps to publish your multi data sources of different cloud providers into the AMG for real time operations observability .
1. Set up the AWS Cloud Watch metrics & Logs data source in AMG
2. Set up the Google Cloud data source in AMG
3. Set up the Microsoft Azure data source in AMG
4. Set up the enterprise data sources available on-premisses in AMG
Prerequisites: setup your workspace for Grafana in AWS Managed Services for Grafana (AMG).
Step 1. Set up the AWS Cloud Watch metrics & Logs data source in AMG
AMG support built-in in plugin for Amazon Cloud Watch data source. Amazon CloudWatch simplifies your existing CloudWatch accounts and manages the configuration of the authentication credentials that required to access CloudWatch. Adding the CloudWatch data source manually support three different authentication methods available. AWS SK Client, Credentials File & Access & secret keys. For more detailed information please refer to the official documentation here.
Along with Amazon CloudWatch, AMG also support more built-in data sources. The full-list of data sources can be found here.
Along with the data sources, AMG also have built-in dashboards with every data sources, which you can import as per your choice.
Step 2. Set up Google Cloud Provider Data Source in AMG
AMG support built-in in plugin for Google Cloud monitor data source. In order to authenticate with the Google Cloud Monitoring API, you must create a Google Cloud Platform Service Account for the project you want to pull the data for. The first step is to upload or paste in the service account key file. For more detailed information please refer to the official documentation here
There are two ways to authenticate the Google Cloud Monitoring plugin
· Upload a Google JWT file
· Automatically retrieve credentials from Google metadata server
The latter option is only available when running Grafana on GCE virtual machine.
Along with the data sources, AMG also have built-in dashboards with every data sources, which you can import as per your choice. With Google Cloud Data sources, following are the pre integrated dashboards available for you to import.
Step 3. Set up Microsoft Azure Data Source in AMG
Likewise the steps we have followed in the previous steps, we will configure the data source for Azure. The azure monitor data source supports multiple services in the Azure cloud:
· Azure Monitor Service: service which is used to provide a single source of monitoring Azure resources
· Application Insights Server: is a service for Application performance Management (APM)
· Azure Log Analytics: provides an access to log data captured by Azure Monitor service
· Use the Application Insights Analytics service: to query Application Insight data using the same query language used by Azure log Analytics.
As per your need, Azure Monitor Data source has options to select your needs and configure your data source accordingly.
1. If you are using Azure Monitor, then you need 4 set of information from the Azure portal. For more information, please find the documentation here.
· Tenant Id (Azure Active Directory -> Properties -> Directory ID)
· Subscription Id (Subscriptions -> Choose subscription -> Overview -> Subscription ID)
· Client Id (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
· Client Secret ( Azure Active Directory -> App Registrations -> Choose your app -> Keys)
1. If you are also using the Azure Log Analytics service, then you need to specify the following two config values (or you can reuse the Client Id and Secret from the previous step).
· Client Id (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
· Client Secret ( Azure Active Directory -> App Registrations -> Choose your app -> Keys -> Create a key -> Use client secret)
1. If you are are using Application Insights, then you need specify the following two pieces of information from the Azure Portal:
· Application ID
· API Key
After the data source is configured, import a custom dashboard to analyze the Azure metrics. You can have the large collection of pre-built in dashboards from Grafana collection
This will import the Azure Virtual Machine dashboard where you can start analyzing the Azure Monitor metrics. In my setup, I have a virtual machine running in the Azure environment.
Step 4. Set up the enterprise data sources available on-premisses in AMG
AMG support many built-in data sources along with the data sources available in Grafana Enterprise edition. However, for this blog we will show you single external data source such as Microsoft SQL Server. Where all the telemetry data from on-premises has been residing into the SQL server. For the complete list of supported data sources, please refer to the official documentation here.
Customers can take advantage of a of a large collection of pre-built Grafana dashboards built by the community that can be easily imported into your Grafana workspace and provide a domain specific quick start to visualizing and investigating observability data for a variety of popular data sources.
Conclusion
With AMG, You can leverage Grafana without having to worry about the hybrid multi-cloud operational observability that involves various measuring, collecting, and analyzing various diagnostics signals from different sources. You can now use Grafana for single pane of glass of your single 360 degree view dashboard without having to worry about the operational management of maintaining infrastructure resources, and let AWS take care of the undifferentiated heavy lifting.
In this blog we have demonstrated how you can use AWS managed services for Grafana to ingest telemetry data in real-time from Hybrid multi-cloud deployments in real-time and how you can visualize this data using an open-source visualizations platform (Grafana). This blog also shows how pre built in data sources and available enterprise grade data sources saves your efforts in integrating observability stack. Leveraging Grafana you can use pre integrated dashboard available with most of the data sources and can create beautiful graphs, insightful logs and valuable alerts in minutes. This enable companies to quickly provision their complete 360 degree view of Hybrid multi-cloud deployments without much worrying about managing the underlying infrastructure using Amazon managed services for Grafana.
