One machine on the internet tried 138 different usernames against four servers last week, 906 times. Nothing on those servers raised a hand about it. The evidence sat in plain text files the whole time, buried in 40,000 lines of login records that nobody was ever going to read.
Every server, firewall and app writes down what it does. Those notes are log files, and on their own they are close to useless, because each one sits on the machine that wrote it in whatever format that machine felt like using. Splunk collects all of them, keeps them in one place, and lets you ask a question across the whole pile in a single line.
Why grep stops working
With one server and one log file you do not need Splunk. You need grep, the command-line tool that prints every line containing a word, and grep is very good at that job. The trouble starts the moment the answer is not inside one file.
Picture a shop that goes slow every Tuesday night. The web servers write one log, the database writes another, the payment service writes a third, and they sit on five machines with five clocks and three date formats.
The grep command will read all of those files, and -c will even count the matching lines. But the files sit on five different machines, so you are running it five times, and grep still cannot group by hour, average a response time, or line up a traffic spike on one server against a slow query on another. You end up with five terminal windows and a guess.
Logs also vanish. Machines rotate them, which means old files get squeezed down and then deleted so the disk does not fill up. The break-in you need to investigate happened five weeks ago. The file that would prove it was deleted four weeks ago.
What Splunk actually does
Splunk reads your log files as they are written, copies every line into storage of its own, and builds a word index across the lot, the way a search engine indexes pages. You never design a table first. For a format Splunk already knows you point it at a file and it starts, and the odd ones need a couple of timestamp and line-splitting settings.
When you search, Splunk splits each line into named values on the fly. A web server entry carries a page name, a response code, a browser and a byte count, and Splunk pulls each one out as a field you can count, sort and average. That step is what turns a wall of text into something you can do arithmetic on.
What you get at the end is one search box covering every line it has ever read, from every machine you pointed it at. The screenshot below is my own install holding 109,864 events from three completely different kinds of log.
Getting your data in
Three routes cover nearly everything. You can upload a file by hand, which is how everyone starts and how you would open a log a customer emailed you. You can point Splunk at a file or folder on its own machine, and it picks up new lines as they land. For every other machine you install a forwarder, a small program whose only job is to read local logs and ship them over.
Forwarders are the normal answer at any real size, because you do not want a full Splunk instance indexing and serving searches on your database server. A forwarder only reads and ships. A forwarder also remembers its place in each file, so restarting it does not re-send yesterday or skip an hour.
Your first search
Most searches start by saying where to look, and index=main is the default store. Add a bare word after that and Splunk behaves like a search engine, handing back any line containing the word.
index=main
index=main error
index=main sourcetype=access_combined_wcookie status=500That third line uses two fields you never had to set up. Splunk writes sourcetype as the label for a kind of log, so access_combined_wcookie means web server traffic. It recognised that format on sight and applied the extractions it already ships for it, one of which is status, the number a web server sends back to say whether a page worked.
Click a field name in the left column and Splunk shows every value it holds with a count and a percentage next to it. Across the week of web traffic I loaded, 86.8% of requests came back as 200, the code for success, and the rest were spread over eight other codes, every one of them a 4xx or 5xx error. That took one click and no configuration.
The pipe is the whole language
Splunk's language is called SPL. Once you can write that first line, one idea carries you through the rest of it. A vertical bar takes the results of whatever is on its left and feeds them into whatever is on its right, so every search is a chain.
index=main sourcetype=access_combined_wcookie status>=500
| stats count by hostThe first line finds every failed request. The second counts them per machine with stats count by host. On my week of data three lines come back, www3 with 785, www1 with 763 and www2 with 540. No single server runs away with it, which makes one sick machine a weaker explanation than something in the application. Check each server's share of the traffic before you settle on that.
Searches you can copy
Almost everything useful is a search plus one summarising command. Every query below ran against real data on my own install, and each answers a question you would otherwise open three files to guess at.
Counting how often each page gets hit tells you where your traffic really goes. On mine the shopping cart took 32% of all requests, which is a strange shape for a shop and worth chasing.
index=main sourcetype=access_combined_wcookie
| top limit=5 uri_pathRaw counts lie, so turn them into a rate. My 2,088 server errors mean nothing until you know they happened across 38,312 requests. That works out at 5.45%, which is far too high to ignore.
index=main sourcetype=access_combined_wcookie
| stats count as total, count(eval(status>=500)) as errors
| eval error_pct=round(errors*100.0/total,2)Watching a number move beats knowing its total. Use timechart to drop results into time slices, so span=4h gives six numbers for a day and shows which parts of it are quiet. Drop to span=1h when the hour itself is the thing you need.
index=main sourcetype=access_combined_wcookie
| timechart span=4h count as requestsWhen Splunk has not pulled out the field you want, rex builds one on the spot from a pattern you supply. This is the search that found the attack in the opening paragraph. It takes every failed login, lifts the address and the username out of the raw text, then counts how many different names each address tried.
index=main sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| rex "for (invalid user )?(?<user>\S+) from"
| stats count as tries, dc(user) as users, dc(host) as servers by src_ip
| sort -triesThe top row came back with 906 tries against 138 usernames across four servers, all from one address. A person who forgot a password does not produce that. Somebody working down a list does. Note what the search never needed, which is a parser written in advance for that username field, or any guess about it back when the data was stored.
From a table to a chart
Most tables Splunk produces turn into a chart by clicking the Visualization tab. The search does not change and you do not pick axes, because the first column becomes the horizontal axis and everything else becomes a series.
A table of counts by response code is a thing you read row by row. The same numbers drawn as columns make the shape obvious from across the room, and most of the value in log searching is spotting the shape that does not belong.
Once a search earns its keep, pin it to a dashboard. A dashboard is a saved page of searches that re-run when you open it, so the numbers are current rather than a screenshot from last month. The one below took about twenty minutes and answers the three questions somebody actually asks in the morning.
Alerts that reach you
A search is only worth so much when somebody has to remember to run it. Save any search as an alert and Splunk runs it on a schedule, then does something when the result crosses a line you set.
An alert needs three settings. Run the failed login search every hour, trigger when the count goes above 50, and send mail or fire a webhook when it does. Turn on throttling as well, or a single loud night will send four hundred messages and teach you to ignore all of them.
One warning before you lean on any of this. Alerting is one of the features the free licence removes. It runs on the 60 day trial and on every paid licence, and it stops the day you switch to the free one.
What Splunk costs
This is where beginners get hurt, so here it is plainly. Splunk does not publish a price for its main log platform. The pricing page lists dollar figures for the monitoring products, starting at $15 per host per month billed annually for infrastructure monitoring, with other lines priced per CPU core or per user. For the log platform itself it asks you to contact sales for a quote.
Ingest pricing charges by how much data you send per day, measured in gigabytes. That sounds fair until you notice it punishes the exact thing the product is good at. Every extra log source, every debug line a developer leaves on, and every chatty firewall rule raises the bill.
So teams end up arguing about what not to collect, which is a terrible way to run a search tool. For the log platform Splunk also sells workload pricing, metered on compute rather than volume, plus an activity-based model. The per-host plans belong to its monitoring products.
You can still learn the core of it for nothing. The Splunk Enterprise download gives you a 60 day trial at 500 MB of data a day, and when the trial ends you can switch to a free licence that keeps the same 500 MB limit and never expires. Everything in this article was done on that download, while the trial was still running.
Read the small print on the free licence before you build anything on it. Alerts stop firing. User accounts and roles stop working, so the login screen disappears and anyone who reaches the web page is admin.
It also runs as a single instance, with no clustering, distributed search or central forwarder management, though forwarders can still ship data into it. Going over the limit three times in 30 days locks searching until you have fewer than three warnings left inside that rolling window. It is a fine place to practise and a bad place to keep secrets.
One more thing worth knowing about who you are buying from. Cisco bought Splunk on 18 March 2024 for about $28 billion, at $157 a share, so the roadmap now belongs to a networking company.
The alternatives, and when they win
Splunk is not the only way to do this, and for a lot of people it is the wrong way. The honest comparison comes down to what you would rather spend, money or your own time.
Elasticsearch with Kibana is the usual free answer. It indexes full text like Splunk does, draws similar charts, and its free tier covers most of what a small team needs, with machine learning and company logins sitting behind a paid subscription. Elastic said in August 2024 that it would add the AGPL as a licence option, and the change landed later that year, adding an Open Source Initiative approved licence alongside the two Elastic already used.
What it costs instead is you. You size the cluster, you tune the memory, you write the parsing rules that Splunk mostly guesses, and you get paged when a node fills its disk. Teams with an engineer who enjoys that work do very well on it. Teams without one quietly lose a month.
Grafana Loki takes the opposite bet. It does not index the contents of your logs at all, only a small set of labels such as which app and which machine a line came from, and it keeps the compressed text in cheap storage. Storage bills drop hard.
You pay for that somewhere else. A search across a wide time range has to read the raw text rather than consult an index, so casual questions get slower as the range grows. If your queries usually start with a known app and a known day, the trade is a bargain.
Datadog and Sumo Logic sell the hosted version of this idea, where nothing is yours to run. Datadog bills per gigabyte taken in and again for the events you keep searchable, and Sumo Logic meters a credit pool that your volume drains, so both land you back in the same argument about what not to collect.
Graylog and OpenSearch cover the run-it-yourself middle ground. Wazuh is worth a look if your only real interest is security alerting, since it ships with detection rules already written.
So why pay for Splunk
Three things justify the money, and none of them is the search box. Splunk gets useful results out of most logs before you write a single parsing rule, and lets you fix the awkward ones afterwards. That sounds like a small favour until you have written those rules by hand for a vendor box that changes its format at every update.
SPL is the second one. It keeps going long after the basics, through joins, statistics and pattern matching, so the hard question you hit three months from now uses the same language as your first search.
Most important, Splunk works out most fields when you search rather than when the data is stored. Only a handful, the time, the host, the file and the log type, are fixed as it lands.
Every system that pins its structure up front makes you decide which questions matter before you have ever asked one. That guess is always wrong once something breaks. Splunk keeps the raw line, so you can invent a field at three in the morning with rex, against data you collected before you knew the field existed.
That is what the money buys, and it is worth it to a team drowning in twelve log formats with an audit deadline. For a hobby server, two apps and a firewall, it is not close, and Loki or Elasticsearch will serve you better.
Either way, install the free copy this weekend and load a week of your own logs. An hour of real searches against data you recognise teaches more than any tutorial, this one included.
