日志处理服务器
When things go south with our applications — as they sometimes do, whether we like it or not — our log files are normally among the first places where we go when we start the troubleshooting process. The big “but” here is that despite the fact that log files contain a wealth of helpful information about events, they are usually extremely difficult to decipher.
当应用程序无法正常运行时(无论是否喜欢，有时有时会如此)，我们的日志文件通常是在开始故障排除过程时首先要去的地方。 这里最大的“但是”是，尽管事实是日志文件包含了大量有关事件的有用信息，但是通常很难解密它们。
A modern web application environment consists of multiple log sources, which collectively output thousands of log lines written in unintelligible machine language. If you, for example, have a LAMP stack set up, then you have PHP, Apache, and MySQL logs to go through. Add system and environment logs into the fray — together with framework-specific logs such as Laravel logs — and you end up with an endless pile of machine data.
现代的Web应用程序环境由多个日志源组成，这些日志源共同输出以难以理解的机器语言编写的数千条日志行。 例如，如果您设置了LAMP堆栈，那么您将需要通过PHP，Apache和MySQL日志。 将系统和环境日志与特定于框架的日志(例如Laravel日志)一起添加到磁盘堆中，最终您将获得无穷无尽的机器数据。
Talk about a needle in a haystack.
谈论大海捞针。
The ELK Stack (Elasticsearch, Logstash, and Kibana) is quickly becoming the most popular way to handle this challenge. Already the most popular open-source log analysis platform — with 500,000 downloads a month, according to Elastic — ELK is a great way to centralize logs from multiple sources, identify correlations, and perform deep-data analysis.
ELK堆栈( Elasticsearch ， Logstash和Kibana )正Swift成为应对这一挑战的最流行方法。 根据Elastic的说法，ELK已经是最受欢迎的开源日志分析平台，每月下载量为500,000，是一种集中来自多个来源的日志，识别关联并执行深度数据分析的好方法。
Elasticsearch is a search-and-analytics engine based on Apache Lucene that allows users to search and analyze large amounts of data in almost real time. Logstash can ingest and forward logs from anywhere to anywhere. Kibana is the stack’s pretty face — a user interface that allows you to query, visualize, and explore Elasticsearch data easily.
Elasticsearch是基于Apache Lucene的搜索和分析引擎，它使用户几乎可以实时搜索和分析大量数据。 Logstash可以从任何地方提取日志并将其转发到任何地方。 Kibana是堆栈的漂亮面Kong-用户界面，可让您轻松查询，可视化和浏览Elasticsearch数据。
This article will describe how to set up the ELK Stack on a local development environment, ship web server logs (Apache logs in this case) into Elasticsearch using Logstash, and then analyze the data in Kibana.
本文将介绍如何在本地开发环境上设置ELK Stack，如何使用Logstash将Web服务器日志(在这种情况下为Apache日志)运送到Elasticsearch中，然后在Kibana中分析数据。
The ELK Stack requires Java 7 and higher (only Oracle’s Java and the OpenJDK are supported), so as an initial step, update your system and run the following:
ELK Stack需要Java 7和更高版本(仅支持Oracle的Java和OpenJDK)，因此，第一步，请更新系统并运行以下命令：
sudo apt-get install default-jreThere are numerous ways of installing the ELK Stack — you can use Docker, Ansible, Vagrant, Microsoft Azure, AWS, or a hosted ELK solution — just take your pick. There is a vast number of tutorials and guides that will help you along the way, one being this ELK Stack guide that we at Logz.io put together.
安装ELK堆栈的方法有很多-您可以选择使用Docker，Ansible，Vagrant，Microsoft Azure，AWS或托管的ELK解决方案。 有大量的教程和指南将为您提供帮助，其中一个就是我们Logz.io整理的ELK Stack指南 。
We’re going to start the installation process with installing Elasticsearch. There are various ways of setting up Elasticsearch but we will use Apt.
我们将通过安装Elasticsearch开始安装过程。 设置Elasticsearch的方法有多种，但我们将使用Apt。
First, download and install Elastic’s public signing key:
首先，下载并安装Elastic的公共签名密钥：
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -Next, save the repository definition to /etc/apt/sources.list.d/elasticsearch-2.x.list:
接下来，将存储库定义保存到/etc/apt/sources.list.d/elasticsearch-2.x.list ：
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.listLast but not least, update the repository cache and install Elasticsearch:
最后但并非最不重要的一点是，更新存储库缓存并安装Elasticsearch：
sudo apt-get update && sudo apt-get install elasticsearchElasticsearch is now installed. Before we continue to the next components, we’re going to tweak the configuration file a bit:
现在已安装Elasticsearch。 在继续下一个组件之前，我们将对配置文件进行一些调整：
sudo nano /etc/elasticsearch/elasticsearch.ymlSome common configurations involve the restriction of external access to Elasticsearch, so data cannot be hacked or deleted via HTTP API:
一些常见的配置涉及外部访问Elasticsearch的限制，因此无法通过HTTP API入侵或删除数据：
network.host: localhostYou can now restart Elasticsearch:
您现在可以重新启动Elasticsearch：
sudo service elasticsearch restartTo verify that Elasticsearch is running properly, query the following URL using the cURL command:
要验证Elasticsearch是否正常运行，请使用cURL命令查询以下URL：
sudo curl 'http://localhost:9200'You should see the following output in your terminal:
您应该在终端中看到以下输出：
{ "name" : "Jebediah Guthrie", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.1", "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", "build_timestamp" : "2016-04-04T12:25:05Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }To make the service start on boot, run:
要使服务在启动时启动，请运行：
sudo update-rc.d elasticsearch defaults 95 10Logstash, the “L” in the “ELK Stack”, is used at the beginning of the log pipeline, ingesting and collecting data before sending it on to Elasticsearch.
Logstash是“ ELK堆栈”中的“ L”，用于日志管道的开头，在将数据发送到Elasticsearch之前先对其进行摄取和收集。
To install Logstash, add the repository definition to your /etc/apt/sources.list file:
要安装Logstash，请将存储库定义添加到您的/etc/apt/sources.list文件中：
echo "deb http://packages.elastic.co/logstash/2.2/debian stable main" | sudo tee -a /etc/apt/sources.listUpdate your system so that the repository will be ready for use and then install Logstash:
更新系统，以便可以使用存储库，然后安装Logstash：
sudo apt-get update && sudo apt-get install logstashWe’ll be returning to Logstash later to configure log shipping into Elasticsearch.
我们稍后将返回Logstash来配置将日志传送到Elasticsearch中。
The final piece of the puzzle is Kibana – the ELK Stack’s pretty face. First, create the Kibana source list:
难题的最后一块是Kibana – ELK Stack的漂亮面Kong。 首先，创建Kibana来源列表：
echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.listThen, update and install Kibana:
然后，更新并安装Kibana：
sudo apt-get update && apt-get install kibanaConfigure the Kibana configuration file at /opt/kibana/config/kibana.yml:
配置在Kibana配置文件/opt/kibana/config/kibana.yml ：
sudo vi /opt/kibana/config/kibana.ymlUncomment the following lines:
取消注释以下行：
server.port: 5601 server.host: “0.0.0.0”Last but not least, start Kibana:
最后但并非最不重要的一点是，启动Kibana：
sudo service kibana startYou can access Kibana in your browser at http://localhost:5601/ (change the URL if you’re using a VM like Homestead Improved to whichever host/port you configured):
您可以在浏览器中通过以下http://localhost:5601/访问Kibana， http://localhost:5601/为http://localhost:5601/ (如果您使用的虚拟机如Homestead Enhanced ，则将URL更改为您配置的主机/端口)：
To start analyzing logs in Kibana, at least one index pattern needs to be defined. An index is how Elasticsearch organizes data, and it can be compared to a database in the world of RDBMS, with mapping defining multiple types.
要开始在Kibana中分析日志，至少需要定义一个索引模式。 索引是Elasticsearch组织数据的方式，并且可以与RDBMS领域中的数据库进行比较，其中映射定义了多种类型。
You will notice that since we have not yet shipped any logs, Kibana is unable to fetch mapping (as indicated by the grey button at the bottom of the page). We will take care of this in the next few steps.
您会注意到，由于我们尚未交付任何日志，因此Kibana无法获取映射(如页面底部的灰色按钮所示)。 在接下来的几个步骤中，我们将对此进行处理。
Tip: By default, Kibana connects to the Elasticsearch instance running on localhost, but you can connect to a different Elasticsearch instance. Simply modify the Elasticsearch URL in the Kibana configuration file that you had edited earlier and then restart Kibana.
提示：默认情况下，Kibana连接到在localhost上运行的Elasticsearch实例，但是您可以连接到其他Elasticsearch实例。 只需在您先前编辑的Kibana配置文件中修改Elasticsearch URL，然后重新启动Kibana。
Our next step is to set up a log pipeline into Elasticsearch for indexing and analysis using Kibana. There are various ways of forwarding data into Elasticsearch, but we’re going to use Logstash.
我们的下一步是建立一个到Elasticsearch的日志管道，以使用Kibana进行索引和分析。 有多种将数据转发到Elasticsearch的方法，但我们将使用Logstash。
Logstash configuration files are written in JSON format and reside in /etc/logstash/conf.d. The configuration consists of three plugin sections: input, filter, and output.
Logstash配置文件以JSON格式编写，并位于/etc/logstash/conf.d 。 该配置包含三个插件部分：输入，过滤器和输出。
Create a configuration file called apache-logs.conf:
创建一个名为apache-logs.conf的配置文件：
sudo vi /etc/logstash/conf.d/apache-logs.confOur first task is to configure the input section, which defines where data is being pulled from.
我们的第一个任务是配置输入部分，该部分定义从何处提取数据。
In this case, we’re going to define the path to our Apache access log, but you could enter a path to any other set of log files (e.g. the path to your PHP error logs).
在这种情况下，我们将定义Apache访问日志的路径，但是您可以输入其他任何日志文件集的路径(例如，PHP错误日志的路径)。
Before doing so, however, I recommend doing some research into supported input plugins and how to define them. In some cases, other log forwarders such as Filebeat and Fluentd are recommended.
但是，在此之前，我建议对支持的输入插件以及如何定义它们进行一些研究。 在某些情况下，建议使用其他日志转发器，例如Filebeat和Fluentd 。
The input configuration:
输入配置：
input { file { path => "/var/log/apache2/access.log" type => "apache-access" } }Our next task is to configure a filter.
我们的下一个任务是配置过滤器。
Filter plugins allow us to take our raw data and try to make sense of it. One of these plugins is grok — a plugin used to derive structure out of unstructured data. Using grok, you can define a search and extract part of your log lines into structured fields.
过滤器插件使我们能够获取原始数据并尝试加以利用。 这些插件之一是grok-一种用于从非结构化数据中派生结构的插件。 使用grok，您可以定义搜索并将部分日志行提取到结构化字段中。
filter { if [type] == "apache-access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } }The last section of the Logstash configuration file is the Output section, which defines the location to where the logs are sent. In our case, it is our local Elasticsearch instance on our localhost:
Logstash配置文件的最后一部分是“输出”部分，它定义了日志发送到的位置。 在我们的例子中，它是本地主机上的本地Elasticsearch实例：
output { elasticsearch {} }That’s it. Once you’re done, start Logstash with the new configuration:
而已。 完成后，使用新配置启动Logstash：
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/apache-logs.confYou should see the following JSON output from Logstash indicating that all is in order:
您应该从Logstash看到以下JSON输出，指示一切正常：
{ "message" => "127.0.0.1 - - [24/Apr/2016:11:41:59 +0000] \"GET / HTTP/1.1\" 200 11764 \"-\" \"curl/7.35.0\"", "@version" => "1", "@timestamp" => "2016-04-24T11:43:34.245Z", "path" => "/var/log/apache2/access.log", "host" => "ip-172-31-46-40", "type" => "apache-access", "clientip" => "127.0.0.1", "ident" => "-", "auth" => "-", "timestamp" => "24/Apr/2016:11:41:59 +0000", "verb" => "GET", "request" => "/", "httpversion" => "1.1", "response" => "200", "bytes" => "11764", "referrer" => "\"-\"", "agent" => "\"curl/7.35.0\"" }Refresh Kibana in your browser, and you’ll notice that the index pattern for our Apache logs was identified:
在浏览器中刷新Kibana，您会注意到已确定我们的Apache日志的索引模式：
Click the Create button, and then select the Discover tab:
单击创建按钮，然后选择发现选项卡：
From this point onwards, Logstash is tailing the Apache access log for messages so that any new entries will be forwarded into Elasticsearch.
从现在开始，Logstash将在Apache访问日志中添加消息，以便所有新条目都将转发到Elasticsearch。
Now that our pipeline is up and running, it’s time to have some fun.
现在我们的管道已经启动并正在运行，现在该找点乐子了。
To make things a bit more interesting, let’s simulate some noise on our web server. To do this I’m going to download some sample Apache logs and insert them into the Apache access log. Logstash is already tailing this log, so these messages will be indexed into Elasticsearch and displayed in Kibana:
为了使事情更有趣，让我们在Web服务器上模拟一些噪音。 为此，我将下载一些示例Apache日志并将其插入Apache访问日志。 Logstash已经在添加此日志，因此这些消息将被索引到Elasticsearch中并显示在Kibana中：
wget http://logz.io/sample-data sudo -i cat /home/ubuntu/sample-data >> /var/log/apache2/access.log exitSearching is the bread and butter of the ELK Stack, and it’s an art unto itself. There is a large amount of documentation available online, but I thought I’d cover the essentials so that you will have a solid base from which to start your exploration work.
搜索是ELK Stack的基础，这本身就是一门艺术。 在线上有大量文档，但是我想我会介绍要点，以便您有一个坚实的基础来开始您的勘探工作。
Let’s start with some simple searches.
让我们从一些简单的搜索开始。
The most basic search is the “free text” search that is performed against all indexed fields. For example, if you’re analyzing web server logs, you could search for a specific browser type (searching is performed using the wide search box at the top of the page):
最基本的搜索是针对所有索引字段执行的“自由文本”搜索。 例如，如果要分析Web服务器日志，则可以搜索特定的浏览器类型(使用页面顶部的宽搜索框执行搜索)：
ChromeIt’s important to note that free text searches are NOT case-sensitive unless you use double quotes, in which case the search results show exact matches to your query.
重要的是要注意，自由文本搜索不区分大小写，除非您使用双引号，在这种情况下，搜索结果显示与查询完全匹配。
“Chrome”Next up are the field-level searches.
接下来是字段级搜索。
To search for a value in a specific field, you need to add the name of the field as a prefix to the value:
要在特定字段中搜索值，您需要添加字段名称作为该值的前缀：
type:apache-accessSay, for example, that you’re looking for a specific web server response. Enter response:200 to limit results to those containing that response.
举例来说，假设您正在寻找特定的Web服务器响应。 输入response:200以将结果限制为包含该响应的结果。
You can also search for a range within a field. If you use brackets [], the results will be inclusive. If you use curly braces {}, the results will exclude the specified values in the query.
您也可以在字段中搜索范围。 如果使用方括号[]，则结果将包含在内。 如果使用大括号{}，则结果将排除查询中的指定值。
Now, it’s time to take it up a notch.
现在，是时候提高自己的水平了。
The next types of searches involve using logical statements. These are quite intuitive but require some finesse because they are extremely syntax-sensitive.
下一类搜索涉及使用逻辑语句。 这些非常直观，但是需要一些技巧，因为它们对语法非常敏感。
These statements include the use of the Boolean operators AND, OR, and NOT:
这些语句包括布尔运算符AND，OR和NOT的使用：
type:apache-access AND (response:400 OR response:500)In the above search, I’m looking for Apache access logs with only a 400 or 500 response. Note the use of parentheses as an example of how more complex queries can be constructed.
在上面的搜索中，我正在寻找仅响应400或500的Apache访问日志。 请注意使用括号作为如何构造更复杂查询的示例。
There are many more search options available (I recommend referring to Logz.io’s Kibana tutorial for more information) such as regular expressions, fuzzy searches, and proximity searches, but once you’ve pinpointed the required data, you can save the search for future reference and as the basis to create Kibana visualizations.
有更多搜索选项可用(我建议参考Logz.io的Kibana教程以获取更多信息)，例如正则表达式，模糊搜索和邻近搜索，但是一旦确定了所需数据，就可以保存搜索以供将来使用参考，并作为创建Kibana可视化的基础。
One of the most prominent features in the ELK Stack in general and Kibana in particular is the ability to create beautiful visualizations with the ingested data. These visualizations can then be aggregated into a dashboard that you can use to get a comprehensive view of all the various log files coming into Elasticsearch.
通常，ELK Stack(尤其是Kibana)中最突出的功能之一就是能够使用所摄取的数据创建精美的可视化效果。 然后，这些可视化内容可以聚合到一个仪表板中，您可以使用该仪表板来全面查看进入Elasticsearch的所有各种日志文件。
To create a visualization, select the Visualize tab in Kibana:
要创建可视化，请在Kibana中选择“可视化”选项卡：
There are a number of visualization types that you can select, and which type you will choose will greatly depend on the purpose and end-result you are trying to achieve. In this case, I’m going to select the good ol’ pie chart.
您可以选择多种可视化类型，并且将选择哪种类型在很大程度上取决于您要达到的目的和最终结果。 在这种情况下，我将选择良好的饼图。
We then have another choice — we can create the visualization from either a saved search or a new search. In this case, we’re going with the latter.
然后，我们有了另一种选择-我们可以从保存的搜索或新的搜索中创建可视化。 在这种情况下，我们将使用后者。
Our next step is to configure the various metrics and aggregations for the graph’s X and Y axes. In this case, we’re going to use the entire index as our search base (by not entering a search query in the search box) and then cross reference the data with browser type: Chrome, Firefox, Internet Explorer, and Safari:
我们的下一步是为图形的X和Y轴配置各种指标和聚合。 在这种情况下，我们将使用整个索引作为搜索基础(不在搜索框中输入搜索查询)，然后交叉引用浏览器类型的数据：Chrome，Firefox，Internet Explorer和Safari：
Once you are finished, save the visualization. You can then add it to a custom dashboard in the Dashboard tab in Kibana.
完成后，保存可视化文件。 然后，您可以将其添加到Kibana中“仪表板”选项卡中的自定义仪表板。
Visualizations are incredibly rich tools to have, and they are the best way to understand the trends within your data.
可视化是非常丰富的工具，它们是了解数据趋势的最佳方法。
The ELK Stack is becoming THE way to analyze and manage logs. The fact that the stack is open source and that it’s backed by a strong community and a fast growing ecosystem is driving its popularity.
ELK堆栈正在成为分析和管理日志的方式。 堆栈是开源的，并且有强大的社区和快速发展的生态系统作为后盾，这推动了其流行。
DevOps is not the sole realm of log analysis, and ELK is being used by developers, sysadmins, SEO experts, and marketers as well. Log-driven development — the development process in which code is monitored using metrics, alerts, and logs — is gaining traction within more and more R&D teams, and it would not be a stretch of the imagination to tie this to the growing popularity of ELK.
DevOps并不是日志分析的唯一领域，开发人员，系统管理员，SEO专家和营销人员也正在使用ELK。 日志驱动的开发(使用度量，警报和日志监控代码的开发过程)在越来越多的研发团队中越来越受欢迎，并且将其与ELK的日益普及联系起来不是想象力的延伸。
Of course, no system is perfect and there are pitfalls that users need to avoid, especially when handling big production operations. But this should not deter you from trying it out, especially because there are numerous sources of information that will guide you through the process.
当然，没有一个系统是完美的，用户需要避免一些陷阱，尤其是在进行大型生产操作时。 但这并不能阻止您尝试一下，尤其是因为有许多信息源可以指导您完成整个过程。
Good luck, and happy indexing!
祝你好运，索引愉快！
This article was peer reviewed by Christopher Thomas, Younes Rafie, and Scott Molinari. Thanks to all of SitePoint’s peer reviewers for making SitePoint content the best it can be!
本文由Christopher Thomas ， Younes Rafie和Scott Molinari 进行了同行评审 。 感谢所有SitePoint的同行评审人员使SitePoint内容达到最佳状态！
翻译自: https://www.sitepoint.com/how-can-the-elk-stack-be-used-to-monitor-php-apps/
日志处理服务器
