Effective log management is essential in today’s complex microservices and distributed systems landscape. Loki, developed by Grafana Labs, is a scalable log aggregation system that integrates seamlessly with Grafana for log visualization and analysis. In this article, we’ll walk through deploying Loki on Kubernetes using local storage and configuring it to be queried from Grafana Cloud. We’ll also set up Grafana Alloy to send logs to our Loki instance.
Prerequisites
Before we begin, make sure you have the following:
- Kubernetes Cluster: A running Kubernetes cluster where you can deploy Loki.
- Grafana Cloud Account: An account on Grafana Cloud with permissions to add data sources.
- kubectl and Helm Installed: Command-line tools to interact with your Kubernetes cluster.
- Domain and SSL Certificate (Optional): If you plan to expose Loki via Ingress with TLS.
Storage Recommendations
- Disk Size: Allocate sufficient disk space based on your log volume. A starting point could be 100GB, but adjust according to your needs.
- IOPS: Use SSDs with high IOPS for better performance, especially if you’re handling a high volume of logs.
- Filesystem: Use a journaling filesystem like ext4 or xfs for reliability.
- Redundancy: Consider RAID configurations (e.g., RAID 1 or RAID 10) for redundancy and performance.
- Monitoring: Monitor disk usage and performance to prevent issues due to resource exhaustion.
Considerations on using K8s over Local Deployment
- Scalability: Kubernetes makes it easy to scale your Loki deployment horizontally by adjusting replicas, allowing you to handle increased load seamlessly.
- High Availability: Kubernetes can automatically restart pods if they fail and distribute them across nodes for fault tolerance.
- Resource Management: Kubernetes manages CPU, memory, and storage resources efficiently, ensuring optimal performance.
- Declarative Configuration: Infrastructure as code allows for consistent deployments and easy rollbacks using manifests and Helm charts.
- Automated Updates: Rolling updates and rollbacks are simplified, reducing downtime during maintenance.
- Service Discovery and Load Balancing: Kubernetes services provide built-in service discovery and load balancing between pods.
- Isolation and Security: Namespaces and network policies help isolate applications and enforce security policies.
- Ecosystem Integration: Kubernetes integrates well with other tools like Prometheus, Grafana Alloy, and CI/CD pipelines.
- Complexity: Kubernetes has a steep learning curve and can be overkill for simple deployments.
- Operational Overhead: Managing a Kubernetes cluster requires additional effort and expertise.
- Resource Consumption: Kubernetes components consume resources, which might not be ideal for small-scale deployments.
- Debugging Challenges: Troubleshooting issues can be more complex due to the distributed nature of Kubernetes.
- Cost: If using managed Kubernetes services, costs can accumulate, especially for small teams or projects.
Step 1: Deploy Loki on K8s
To reduce complexity, we’ll leverage the official Helm chart provided by Grafana Labs. This simplifies the deployment process significantly.
Add the Grafana Helm Repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
Install Loki using Helm
We will use the default values for a simple setup. For more advanced configurations, refer to the official documentation.
helm upgrade --install loki grafana/loki --namespace loki-stack --create-namespace
This command installs Loki in the loki-stack
namespace. By default, it uses a filesystem storage backend, storing data locally within the pod’s file system.
Configure Persistent Storage (Optional but Recommended)
To ensure data persistence across pod restarts or upgrades, configure a PersistentVolumeClaim (PVC).
Create a custom values.yaml
file:
persistence:
enabled: true
storageClassName: "fast-storage" # Replace with your storage class
accessModes:
- ReadWriteOnce
size: 100Gi
Install Loki with the custom values:
helm upgrade --install loki grafana/loki --namespace loki-stack -f values.yaml
Step 2: Configure Grafana Alloy to Send Logs to Loki
Grafana Alloy collects logs from your environment and forwards them to Loki. We’ll set up Alloy to send logs to the Loki instance we’ve deployed.
Install Grafana Alloy
Grafana Alloy can be installed on various platforms. Follow the Grafana Documentation to install it on your operating system.
Create a Configuration File to send logs to your Loki
Create a file named config.alloy within your current working directory.
The 1st component is to locate the logs. So we will add that to the config:
local.file_match "local_files" {
path_targets = [{"__path__" = "/var/log/*.log"}]
sync_period = "5s"
}
This configuration creates a local.file_match component named local_files
which does the following:
- It tells Alloy which files to source.
- It checks for new files every 5 seconds.
The 2nd component we need is to tell alloy to scrape the logs:
loki.source.file "log_scrape" {
targets = local.file_match.local_files.targets
forward_to = [loki.process.filter_logs.receiver]
tail_from_end = true
}
This configuration creates a loki.source.file
component named log_scrape
which does the following:
- It connects to the
local_files
component as its source or target. - It forwards the logs it scrapes to the receiver of another component called
filter_logs
. - It provides extra attributes and options to tail the log files from the end so you don’t ingest the entire log file history.
Now, we can go directly for the logs writing, however, normally, you would want to avoid writing to much data that has no relevance to your Loki server, so you would also need a log filtering component. So, our 3rd component will be to filter logs:
loki.process "filter_logs" {
stage.drop {
source = ""
expression = ".*Connection closed by authenticating user root"
drop_counter_reason = "noisy"
}
forward_to = [loki.write.grafana_loki.receiver]
}
This configuration creates a loki.process
component named filter_logs
which does the following:
- It receives scraped log entries from the default
log_scrape
component. - It uses the
stage.drop
block to define what to drop from the scraped logs. - It uses the
expression
parameter to identify the specific log entries to drop. - It uses an optional string label
drop_counter_reason
to show the reason for dropping the log entries. - It forwards the processed logs to the receiver of another component called
grafana_loki
.
The final part is about writing those logs, finally, to Loki.
loki.write "grafana_loki" {
endpoint {
url = "http://localhost:3100/loki/api/v1/push"
// basic_auth {
// username = "admin"
// password = "admin"
// }
}
}
Note: You don’t have to configure authentication if running Loki as a local docker container.
Run Grafana Alloy
Run in Foreground (For Testing)
./alloy --config.file=config.alloy
Run as a Service
Create /etc/systemd/system/grafana-alloy.service
:
[Unit]
Description=Grafana Alloy
After=network.target
[Service]
User=alloy
Group=alloy
Type=simple
ExecStart=/usr/local/bin/alloy --config.file=/etc/alloy/config.alloy
Restart=on-failure
[Install]
WantedBy=multi-user.target
Set up the Alloy user and directories:
sudo useradd -r -s /bin/false alloy
sudo mkdir -p /etc/alloy /tmp/alloy/{data,positions}
sudo chown -R alloy:alloy /etc/alloy /tmp/alloy
Start the service:
sudo systemctl daemon-reload
sudo systemctl enable grafana-alloy.service
sudo systemctl start grafana-alloy.service
Step 3: Expose Loki for Grafana Cloud
Grafana Cloud needs to access your Loki instance. You have several options to expose Loki securely.
In your values.yaml
, add:
service:
type: LoadBalancer
Update the deployment:
helm upgrade --install loki grafana/loki --namespace loki-stack -f values.yaml
This will provision a LoadBalancer service, and you can use the external IP to access Loki.
In your values.yaml
, add:
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
loki.yourdomain.com
tls:
secretName: loki-tls
hosts:
loki.yourdomain.com
Make sure to:
- Set up DNS records pointing
loki.yourdomain.com
to your ingress controller. - Install and configure cert-manager for SSL certificates.
If exposing Loki over the internet isn’t desirable, you can use Private Data Source Connect (PDC) offered by Grafana Cloud. PDC establishes a secure tunnel between your Grafana Cloud instance and your private data sources.
What is Private Data Source Connect?
Private Data Source Connect allows you to connect Grafana Cloud to data sources that are not publicly accessible. It uses a lightweight agent that runs in your environment and securely proxies requests from Grafana Cloud to your data source.
Steps to Set Up PDC with Loki
- Enable PDC in Grafana Cloud:
- Log in to your Grafana Cloud account.
- Navigate to Connections > Private Data Sources.
- Click Set up Private Data Source Connect.
- Install the PDC Agent:
- Follow the instructions to download and install the PDC agent.
- You can deploy the agent on a local machine or within your Kubernetes cluster for scalability.
pdc-agent.yaml
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana-pdc-agent
namespace: loki-stack
spec:
replicas: 1
selector:
matchLabels:
app: grafana-pdc-agent
template:
metadata:
labels:
app: grafana-pdc-agent
spec:
containers:
- name: pdc-agent
image: grafana/pdc-agent:latest
env:
- name: PDC_AGENT_KEY
value: "<your_pdc_agent_key>"
ports:
- containerPort: 3000
Replace <your_pdc_agent_key>
with the key provided by Grafana Cloud.
Deploy the agent:
kubectl apply -f pdc-agent.yaml
3. Configure Loki as a Private Data Source:
- In Grafana Cloud, add a new data source.
- Select Loki.
- In the URL field, enter the internal service URL of Loki (e.g.,
http://loki.loki-stack.svc.cluster.local:3100
). - Under Access, select Server.
- Enable Private Data Source Connect.
- Save and test the connection.
Advantages of Using PDC
- Security: Your data source remains within your private network.
- No Public Exposure: No need to expose Loki over the internet.
- Scalability: By deploying the PDC agent on Kubernetes, you can scale it as needed.
Considerations
- Network Configuration: Ensure the PDC agent can communicate with Loki internally.
- Resource Usage: The PDC agent consumes resources; monitor its usage and scale accordingly.
Step 4: Configure Grafana Cloud to Query Loki
If you used Option 1 or Option 2 in Step 2, follow these steps:
- Log in to Grafana Cloud and navigate to your Grafana instance.
- Add a New Data Source:
- Go to Configuration > Data sources > Add data source.
- Select Loki.
- Configure the Data Source:
- URL: Use the public URL of your Loki instance (e.g.,
https://loki.yourdomain.com
). - Access: Server.
- Basic Auth: Enable if you’ve set up authentication on Loki.
- Credentials: Enter your username and password if required.
- URL: Use the public URL of your Loki instance (e.g.,
- Save & Test:Click Save & Test to verify the connection.
Step 5: Verify Logs in Grafana
- In your Grafana instance, go to Explore.
- Choose the Loki data source you configured earlier.
- Run a Log Query: <code>{job=”varlogs”}</code>
- You should see logs from your system appearing in Grafana.
Conclusion
By following these steps, you’ve successfully deployed Loki on Kubernetes using local storage. You’ve also configured Grafana Alloy to collect and send logs to Loki and set up Grafana Cloud to query and visualize those logs using Private Data Source Connect. This setup provides a robust and secure logging solution that scales with your infrastructure.
Next Steps:
- Monitor Storage Usage: Make sure to keep an eye on disk space and performance to avoid running out of storage.
- Explore LogQL: Learn Loki’s query language to get the most out of your logs.
- Set Up Dashboards: Create dashboards in Grafana to visualize log data.
- Implement Alerting: Configure alerts based on log patterns or metrics.
- Scale as Needed: Adjust the replicas of your Loki and PDC agent deployments to handle increased load.
Leave a Reply
You must be logged in to post a comment.