- Article
- 9 minutes to read
The Custom Logs data source for the Log Analytics agent in Azure Monitor allows you to collect events from text files on both Windows and Linux computers. Many applications log information to text files instead of standard logging services, such as Windows Event log or Syslog. After the data is collected, you can either parse it into individual fields in your queries or extract it during collection to individual fields.
Important
This article describes how to collect a text log with the Log Analytics agent. If you're using the Azure Monitor agent, then see Collect text logs with Azure Monitor Agent.
Important
The legacy Log Analytics agent will be deprecated by August 2024. Migrate to Azure Monitor agent before August 2024 to continue ingesting data.
The log files to be collected must match the following criteria:
The log must either have a single entry per line or use a timestamp matching one of the following formats at the start of each entry:
YYYY-MM-DD HH:MM:SS
M/D/YYYY HH:MM:SS AM/PM
Mon DD, YYYY HH:MM:SS
yyMMdd HH:mm:ss
ddMMyy HH:mm:ss
MMM d hh:mm:ss
dd/MMM/yyyy:HH:mm:ss zzz
yyyy-MM-ddTHH:mm:ssKThe log file must not allow circular logging. This behavior is log rotation where the file is overwritten with new entries or the file is renamed and the same file name is reused for continued logging.
The log file must use ASCII or UTF-8 encoding. Other formats such as UTF-16 aren't supported.
For Linux, time zone conversion isn't supported for time stamps in the logs.
As a best practice, the log file should include the date and time that it was created to prevent log rotation overwriting or renaming.
Note
If there are duplicate entries in the log file, Azure Monitor will collect them. The query results that are generated will be inconsistent. The filter results will show more events than the result count. You must validate the log to determine if the application that creates it is causing this behavior. Address the issue, if possible, before you create the custom log collection definition.
A Log Analytics workspace supports the following limits:
- Only 500 custom logs can be created.
- A table only supports up to 500 columns.
- The maximum number of characters for the column name is 500.
Important
Custom log collection requires that the application writing the log file flushes the log content to the disk periodically. This is because the custom log collection relies on filesystem change notifications for the log file being tracked.
Define a custom log
Use the following procedure to define a custom log file. Scroll to the end of this article for a walkthrough of a sample of adding a custom log.
Open the Custom Log wizard
The Custom Log wizard runs in the Azure portal and allows you to define a new custom log to collect.
- In the Azure portal, select Log Analytics workspaces > your workspace > Settings.
- Select Custom logs.
- By default, all configuration changes are automatically pushed to all agents. For Linux agents, a configuration file is sent to the Fluentd data collector.
- Select Add to open the Custom Log wizard.
Upload and parse a sample log
To start, upload a sample of the custom log. The wizard will parse and display the entries in this file for you to validate. Azure Monitor will use the delimiter that you specify to identify each record.
New Line is the default delimiter and is used for log files that have a single entry per line. If the line starts with a date and time in one of the available formats, you can specify a Timestamp delimiter, which supports entries that span more than one line.
If a timestamp delimiter is used, the TimeGenerated property of each record stored in Azure Monitor will be populated with the date and time specified for that entry in the log file. If a new line delimiter is used, TimeGenerated is populated with the date and time when Azure Monitor collected the entry.
- Select Browse and browse to a sample file. This button might be labeled Choose File in some browsers.
- Select Next.
- The Custom Log wizard uploads the file and lists the records that it identifies.
- Change the delimiter that's used to identify a new record. Select the delimiter that best identifies the records in your log file.
- Select Next.
Add log collection paths
You must define one or more paths on the agent where it can locate the custom log. You can either provide a specific path and name for the log file or you can specify a path with a wildcard for the name. This step supports applications that create a new file each day or when one file reaches a certain size. You can also provide multiple paths for a single log file.
For example, an application might create a log file each day with the date included in the name as in log20100316.txt. A pattern for such a log might be log*.txt, which would apply to any log file following the application's naming scheme.
The following table provides examples of valid patterns to specify different log files.
| Description | Path |
|---|---|
| All files in C:\Logs with .txt extension on the Windows agent | C:\Logs\*.txt |
| All files in C:\Logs with a name starting with log and a .txt extension on the Windows agent | C:\Logs\log*.txt |
| All files in /var/log/audit with .txt extension on the Linux agent | /var/log/audit/*.txt |
| All files in /var/log/audit with a name starting with log and a .txt extension on the Linux agent | /var/log/audit/log*.txt |
- Select Windows or Linux to specify which path format you're adding.
- Enter the path and select the + button.
- Repeat the process for any more paths.
Provide a name and description for the log
The name that you specify will be used for the log type as described. It will always end with _CL to distinguish it as a custom log.
- Enter a name for the log. The _CL suffix is automatically provided.
- Add an optional Description.
- Select Next to save the custom log definition.
Validate that the custom logs are being collected
It might take up to an hour for the initial data from a new custom log to appear in Azure Monitor. Azure Monitor will start collecting entries from the logs found in the path you specified from the point that you defined the custom log. It won't retain the entries that you uploaded during the custom log creation. It will collect already existing entries in the log files that it locates.
After Azure Monitor starts collecting from the custom log, its records will be available with a log query. Use the name that you gave the custom log as the Type in your query.
Note
If the RawData property is missing from the query, you might need to close and reopen your browser.
Parse the custom log entries
The entire log entry will be stored in a single property called RawData. You'll most likely want to separate the different pieces of information in each entry into individual properties for each record. For options on parsing RawData into multiple properties, see Parse text data in Azure Monitor.
Remove a custom log
Use the following process in the Azure portal to remove a custom log that you previously defined.
- From the Data menu in the Advanced Settings for your workspace, select Custom Logs to list all your custom logs.
- Select Remove next to the custom log to remove the log.
Data collection
Azure Monitor collects new entries from each custom log approximately every 5 minutes. The agent records its place in each log file that it collects from. If the agent goes offline for a period of time, Azure Monitor collects entries from where it last left off, even if those entries were created while the agent was offline.
The entire contents of the log entry are written to a single property called RawData. For methods to parse each imported log entry into multiple properties, see Parse text data in Azure Monitor.
Custom log record properties
Custom log records have a type with the log name that you provide and the properties in the following table.
| Property | Description |
|---|---|
| TimeGenerated | Date and time that the record was collected by Azure Monitor. If the log uses a time-based delimiter, this is the time collected from the entry. |
| SourceSystem | Type of agent the record was collected from. OpsManager – Windows agent, either direct connect or System Center Operations Manager Linux – All Linux agents |
| RawData | Full text of the collected entry. You'll most likely want to parse this data into individual properties. |
| ManagementGroupName | Name of the management group for System Center Operations Manager agents. For other agents, this name is AOI-<workspace ID>. |
Sample walkthrough of adding a custom log
The following section walks through an example of creating a custom log. The sample log being collected has a single entry on each line starting with a date and time and then comma-delimited fields for code, status, and message. Several sample entries are shown.
2019-08-27 01:34:36 207,Success,Client 05a26a97-272a-4bc9-8f64-269d154b0e39 connected2019-08-27 01:33:33 208,Warning,Client ec53d95c-1c88-41ae-8174-92104212de5d disconnected2019-08-27 01:35:44 209,Success,Transaction 10d65890-b003-48f8-9cfc-9c74b51189c8 succeeded2019-08-27 01:38:22 302,Error,Application could not connect to database2019-08-27 01:31:34 303,Error,Application lost connection to databaseUpload and parse a sample log
We provide one of the log files and can see the events that it will be collecting. In this case, New line is a sufficient delimiter. If a single entry in the log could span multiple lines though, a timestamp delimiter would need to be used.
Add log collection paths
The log files will be located in C:\MyApp\Logs. A new file will be created each day with a name that includes the date in the pattern appYYYYMMDD.log. A sufficient pattern for this log would be C:\MyApp\Logs\*.log.
Provide a name and description for the log
We use a name of MyApp_CL and type in a Description.
Validate that the custom logs are being collected
We use a simple query of MyApp_CL to return all records from the collected log.
Alternatives to custom logs
While custom logs are useful if your data fits the criteria listed, there are cases where you need another strategy:
- The data doesn't fit the required structure, such as having the timestamp in a different format.
- The log file doesn't adhere to requirements such as file encoding or an unsupported folder structure.
- The data requires preprocessing or filtering before collection.
In the cases where your data can't be collected with custom logs, consider the following alternate strategies:
- Use a custom script or other method to write data to Windows Events or Syslog, which are collected by Azure Monitor.
- Send the data directly to Azure Monitor by using HTTP Data Collector API.
Next steps
- See Parse text data in Azure Monitor for methods to parse each imported log entry into multiple properties.
- Learn about log queries to analyze the data collected from data sources and solutions.
FAQs
How do I check logs on an Azure Log Analytics? ›
- Sign in to the Azure portal.
- Select Azure Active Directory, and then select Logs from the Monitoring section to open your Log Analytics workspace. The workspace will open with a default query.
Azure Monitor stores these fundamental types of data: metrics, logs, traces, and changes. The sources of monitoring data that populate these data stores are on the left. The different functions that Azure Monitor performs with this collected data are on the right. This includes such actions as analysis, alerting.
Is there a maximum amount of data that I can collect in Azure Monitor? ›...
Ingestion.
| Limit | Value |
|---|---|
| Data ingestion per minute to a data collection endpoint | 50 GB This limit can't be increased. |
The Azure Monitor activity log is a platform log in Azure that provides insight into subscription-level events. The activity log includes information like when a resource is modified or a virtual machine is started. You can view the activity log in the Azure portal or retrieve entries with PowerShell and the Azure CLI.
How do I collect Logs from Azure? ›Open the Custom Log wizard
The Custom Log wizard runs in the Azure portal and allows you to define a new custom log to collect. In the Azure portal, select Log Analytics workspaces > your workspace > Settings. Select Custom logs. By default, all configuration changes are automatically pushed to all agents.
...
The article shows you how to:
- Understand query structure.
- Sort query results.
- Filter query results.
- Specify a time range.
- Select which fields to include in the results.
- Define and use custom fields.
- Aggregate and group results.
Some of the key takeaways of Azure Monitor for your AZ-900 exam are, you can send alerts to Azure Active Directory groups and users, and Azure Monitor can trigger alerts based on data in an Azure Log Analytics workspace.
What can I Monitor with Azure Monitor? ›Azure administrators implement, manage, and monitor an organization's Microsoft Azure environment, including virtual networks, storage, compute, identity, security, and governance.
What can be monitored using Azure Monitor? ›Collect, analyze, and act on telemetry data from your cloud and hybrid environments. Azure Monitor supports your operations at scale by helping you maximize the performance and availability of your resources and proactively identify problems.
What is the daily limit for Log Analytics in Azure? ›The maximum cap for an Application Insights classic resource is 1,000 GB/day unless you request a higher maximum for a high-traffic application. When you create a resource in the Azure portal, the daily cap is set to 100 GB/day.
What is the limit of Log Analytics query? ›
Data limits
Maximum of 30 MB per post to Log Analytics Data Collector API. This is a size limit for a single post. If the data from a single post that exceeds 30 MB, you should split the data up to smaller sized chunks and send them concurrently. Maximum of 32 KB limit for field values.
Storage Analytics has a 20 TB limit on the amount of stored data that is independent of the total limit for your storage account.
How do you collect logs? ›Select the hosts in the host table. Click Actions > Collect Log Files. Click Advanced Options and choose the options for the log file collection. Important: Changed in 7.4.
How does Azure monitor collect data? ›Where does Azure Monitor get its data? Azure Monitor collects data from various sources including logs and metrics from Azure platform and resources, custom applications, and agents running on virtual machines.
What is the difference between Azure Monitor and Log Analytics? ›Azure Monitor builds on top of Azure Log Analytics, the platform service that gathers log and metrics data from all your resources. The easiest way to think about Azure Monitor vs Log Analytics is that Azure Monitor is the marketing name, whereas Azure Log Analytics is the technology that powers it.
How does Azure Monitor organize log data for queries? ›Azure Monitor Logs is based on Azure Data Explorer, and log queries are written by using the same Kusto Query Language (KQL). This rich language is designed to be easy to read and author, so you should be able to start writing queries with some basic guidance.
How to query Azure Storage Logs in Azure Monitor Log Analytics? ›- Enable Azure Storage Diagnostics logs.
- Create a Log analytics workspace.
- Load the Azure Storage diagnostic logs into Log Analytics.
- Query the diagnostic data in Azure Log Analytics.
- Conclusion.
From the Monitoring section of your function app in the Azure portal, select Diagnostic settings, and then select Add diagnostic setting. In the Diagnostics settings page, under Category details and log, choose FunctionAppLogs. The FunctionAppLogs table contains the desired logs.
What should you use to create an activity log alert in Azure Monitor? ›Activity log alert rules are Azure resources, so they can be created by using an Azure Resource Manager template. They also can be created, updated, or deleted in the Azure portal. An activity log alert only monitors events in the subscription in which the alert is created.
Can Azure Monitor trigger alerts? ›Alerts help you detect and address issues before users notice them by proactively notifying you when Azure Monitor data indicates there might be a problem with your infrastructure or application. You can alert on any metric or log data source in the Azure Monitor data platform. This diagram shows you how alerts work.
What is Log Analytics in Azure Monitor? ›
Log Analytics is a tool in the Azure portal to edit and run log queries from data collected by Azure Monitor logs and interactively analyze their results. You can use Log Analytics queries to retrieve records that match particular criteria, identify trends, analyze patterns, and provide various insights into your data.
What are the three main functions of Azure monitor? ›Microsoft combined three unique services—Azure Monitor, Log Analytics, and Application Insights—under the umbrella of Azure Monitor to provide powerful end-to-end monitoring of your applications and the components they rely on. Log Analytics and Application Insights are now features of Azure Monitor.
How long does Azure monitor keep logs? ›You can keep data in interactive retention between 4 and 730 days. You can set the archive period for a total retention time of up to 2,556 days (seven years). To set the retention and archive duration for a table in the Azure portal: From the Log Analytics workspaces menu, select Tables.
What are the two main kinds of data Azure monitor works with? ›Overall, Azure Monitor is using two fundamental types of data to work. Those are metrics and logs the tool is collecting into data stores to perform actions such as analysis, alert generation and streaming to connected external systems.
Where are Azure monitor logs stored? ›Logs in Azure Monitor are stored in a Log Analytics workspace that's based on Azure Data Explorer, which provides a powerful analysis engine and rich query language.
How does Azure monitor work? ›Azure Monitor receives data from target resources like applications, operating systems, Azure resources, Azure subscriptions, and Azure tenants. The nature of the resource defines which data types are available. A data type will be a metric, a log, or both a metric and a log.
Where can I see logs in Azure? ›Open the Azure portal in a web browser. Filter the list of resource by the resource group, rg-demo-vm-eastus . Select the demoWebAppMonitor resource. Select the Monitoring section's Logs item.
How do I view Azure network logs? ›Accessing Logs
For viewing the logs, you can either use the Azure Portal or use the Microsoft Azure Storage Explorer. View the insights-logs-networksecuritygroupflowevent container in the configured storage account. Access the PT1H. json file.
On the Azure portal, navigate to the NSG Flow Logs section in Network Watcher. Then click the name of the NSG. This will bring up the settings pane for the Flow log.
Where are Azure activity logs stored? ›View platform logs
View the activity log in the Azure portal and access events from PowerShell and the Azure CLI. See View the activity log for details. View Azure AD security and activity reports in the Azure portal.
How many types of logs are in Azure? ›
| Log category | Log type |
|---|---|
| Virtual machines and cloud services | Windows Event Log service and Linux Syslog |
| Azure Storage Analytics | Storage logging, provides metrics data for a storage account |
| Network security group (NSG) flow logs | JSON format, shows outbound and inbound flows on a per-rule basis |
Start > Control Panel > System and Security > Administrative Tools > Event Viewer. In event viewer select the type of log that you want to review. Windows stores five types of event logs: application, security, setup, system and forwarded events.
How do I check audit logs in Azure portal? ›- Open the Azure portal.
- Search Azure Active Directory and select it.
- Scroll down panel on the left side of the screen and navigate to Manage.
- Select Users tab.
- Now click on Audit Logs under Activity.
- UserManagement is the pre-selected Category.
Azure Monitor Logs gives you the ability to consolidate logs from different resources in the same workspace, where it can be analyzed with queries to quickly retrieve, consolidate, and analyze collected data.
What are log signals monitored Azure? ›A log alert rule monitors a resource by using a Log Analytics query to evaluate resource logs at a set frequency. If the conditions are met, an alert is fired. Because you can use Log Analytics queries, you can perform advanced logic operations on your data and use the robust KQL features to manipulate log data.
How do I view logs in Azure app? ›To stream logs in the Azure portal, navigate to your app and select Log stream.
How do I send Azure audit logs to Log Analytics? ›On the Diagnostic settings page, click Add diagnostic setting. Under Category details, select AuditLogs and SigninLogs. Under Destination details, select Send to Log Analytics, and then select your new log analytics workspace. Click Save.
How to query Azure storage logs in Azure Monitor Log Analytics? ›- Enable Azure Storage Diagnostics logs.
- Create a Log analytics workspace.
- Load the Azure Storage diagnostic logs into Log Analytics.
- Query the diagnostic data in Azure Log Analytics.
- Conclusion.
Activity logs also record Service Health events. Resource logs capture operations performed within an Azure resource (i.e., operations coming from the data plane), such as querying a database or writing to a storage bucket.