« Back to Index

Different Linux utility commands (e.g. top, ps, strace, lsof, netstat, ifconfig, iftop, iptraf, tcpdump, wireshark)

View original Gist on GitHub

1. linux utilities.md

Start up a container (whichever Linux flavour takes your fancy):

docker run -it ubuntu /bin/bash
docker run -it centos /bin/bash

OSI Model

Layer Protocols Description
7. Application HTTP, FTP, SMTP Window for user app processes
6. Presentation JPEG, GIF, MPEG Format the data to be presented to the Application layer (network translator)
5. Session RPC, SQL, NFS Allow session establishment between processes running on different stations
4. Transport TCP, UDP, SPX Flow control, ensures all messages are delivered error-free, in sequence, no losses or duplications
3. Network IP, IPX, ICMP Routers control operation of subnet, deciding physical path data takes
2. Data Link PPP/SLIP Provides error-free transfer of data over physical layer
1. Physical Hub Physical structure (cables, hubs etc)

ifconfig.md

## ifconfig

The ifconfig command is used to configure a network interface.

For a breakdown of the following ouput see: http://www.aboutlinux.info/2006/11/ifconfig-dissected-and-demystified.html

If you don’t want to configure a network interface then running the command without any arguments will display existing network interfaces.

eth0      Link encap:Ethernet  HWaddr 0A:05:1E:A5:6F:FF  
          inet addr:10.6.4.51  Bcast:10.6.7.255  Mask:255.255.248.0
          inet6 addr: fe80::805:1eff:fea5:6fff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:8776319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4212889 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10239965628 (9.5 GiB)  TX bytes:10967533931 (10.2 GiB)
          Interrupt:155 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:341240 errors:0 dropped:0 overruns:0 frame:0
          TX packets:341240 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:31612016 (30.1 MiB)  TX bytes:31612016 (30.1 MiB)

We can see above that we have a single Ethernet card (eth0) and a loop back interface (lo)

In newer Linux OS’ eth<n> is replaced by p2p<n>

For more information on Network ips and how they’re created then read: https://gist.github.com/Integralist/cff468ba808fbca09602

iftop.md

iftop

iftop does for network usage what top(1) does for CPU usage. It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts. Handy for answering the question “why is our ADSL link so slow?”

To install (Ubuntu):

apt-get update
apt-get install iftop

To install (CentOS):

yum install wget libpcap -y
wget http://pkgs.repoforge.org/iftop/iftop-0.17-1.el6.rf.x86_64.rpm
rpm -ivh iftop-0.17-1.el6.rf.x86_64.rpm

iptraf.md

iptraf

The iptraf tool monitors Inbound and Outbound network traffic passing through a network interface.

We can monitor various connections like TCP, UDP, ICMP, non-IP counts and also Ethernet load information.

To install:

yum install iptraf     # CentOS
apt-get install iptraf # Ubuntu

Of particular interest is the menu Statistical breakdowns... > By TCP/UDP port > eth0 which shows packets for all TCP and UDP ports such as 22 for SSH and 443 for HTTPS.

lsof.md

lsof

An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket)

netstat.md

netstat

netstat (network statistics) is a command-line tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface and network protocol statistics

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name   
tcp        0      0 0.0.0.0:8126                0.0.0.0:*                   LISTEN      0          8934       1071/statsd         
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      498        10087      1355/puma 2.14.0 (t 
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      0          8763       1196/sshd           
tcp        0      0 127.0.0.1:24220             0.0.0.0:*                   LISTEN      497        9296       1307/ruby           
tcp        0      0 10.6.4.51:57228             10.6.31.176:6379            ESTABLISHED 498        218757     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:57224             10.6.31.176:6379            ESTABLISHED 498        218743     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:8080              10.6.8.80:48205             ESTABLISHED 498        229190     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:57231             10.6.31.176:6379            ESTABLISHED 498        218766     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:57225             10.6.31.176:6379            ESTABLISHED 498        218747     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:8080              10.6.6.76:51764             ESTABLISHED 498        10111      1355/puma 2.14.0 (t 
tcp       53      0 10.6.4.51:56870             54.231.142.40:443           ESTABLISHED 497        229141     1307/ruby           
tcp        1      0 127.0.0.1:34704             127.0.0.1:8080              CLOSE_WAIT  48         226010     20286/httpd

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.6.0.0        *               255.255.248.0   U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
default         ip-xx-x-x-x.eu- 0.0.0.0         UG        0 0          0 eth0

ps.md

ps

a needs to be used with x to give you all processes
u provides additional output such as CPU and MEM
so axu would be more accurate, but I’m more used to aux

strace.md

strace

strace monitors interactions between processes, such as: system calls, signal deliveries, and changes of process state

Install strace (Ubuntu):

apt-get install man -y
apt-get install strace
man strace

Install strace (CentOS):

yum install strace -y

With Docker:
--privileged required for ptrace to be allowed
--security-opt seccomp:unconfined is an alternative

strace <binary_executable>

The end of the strace output will be the output of the specified binary executable.

Filter results using an ‘expression’:

strace -e trace=open ls

specify multiple system calls with comma-separated list
-e trace=open,read

Attach strace to a running process:

strace -p <process_id>

ps aux to locate process id

Be careful with backgrounded processes. If you attach to a backgrounded process running in the same shell instance as your strace execution, then you’ll be locked up.

With Docker you can fix this by executing the following within a new shell:

docker exec -it <container_id> /bin/bash

Then kill -9 <strace_pid|application_pid>

Add a timestamp to your output using -t

Statistical summary -c displays output in a graphical table:

output with -c comes after the binary’s output

Redirect and pipe output:

strace php 2>&1 | grep php.ini

tcpdump.md

## tcpdump

tcpdump is a powerful and widely used command-line packets sniffer or package analyzer tool which is used to capture or filter TCP/IP packets that received or transferred over a network on a specific interface

We can save the output into a pcap file format, that can be viewed by either tcpdump itself or via an open source GUI based tool called Wireshark (Network Protocol Analyzier) that reads tcpdump pcap format files

To install:

yum install tcpdump -y     # CentOS
apt-get install tcpdump -y # Ubuntu

Usage

SCP file from AWS remote to local (run on local machine)
scp -v -r <user_name>@<ip>,eu-west-1:<remote_path> <local_path>
Make sure the local directory exists

Flags

Note: FIN wont necessarily be the last packet sent. For example…

telnet.md

telnet

Telnet is a Network Protocol and the tool which uses that protocol (i.e telnet) is also known as Telnet.

The telnet utility is used for interactive communication to a remote/extern host on a given port. Once the connection to the remote host is established, an HTTP request can be send to the host by typing it in the prompt.

The following example shows that the google domain carries out a 302 redirect:

$ telnet www.google.com 80
Trying 87.237.19.30...
Connected to www.google.com.
Escape character is '^]'.
GET #q=cars HTTP/1.1                   

HTTP/1.1 302 Found
Location: http://www.google.co.uk/?gws_rd=cr&ei=k7BjV-GbFOLOgAbd3JbADw#q=cars
Cache-Control: private
Content-Type: text/html; charset=UTF-8
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Date: Fri, 17 Jun 2016 08:10:59 GMT
Server: gws
Content-Length: 268
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=80=R8K0WhuF432ccZzjpchtEPHx-vv1n-9tuoe8P6V2yyNC2h1sd_JB7Q1afFZPo5W9MjvP8UL1ZZ_8UQDHeb3OpGDRNlSNfPkJWqE9JKa9hAJG02wlk7s8eIRy786p7-8U; expires=Sat, 17-Dec-2016 08:10:59 GMT; path=/; domain=.google.com; HttpOnly

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/?gws_rd=cr&amp;ei=k7BjV-GbFOLOgAbd3JbADw#q=cars">here</A>.
</BODY></HTML>
Connection closed by foreign host.

The BBC site gives us a 404 not found:

$ telnet www.bbc.co.uk 80
Trying 212.58.244.66...
Connected to www.bbc.net.uk.
Escape character is '^]'.
GET /news HTTP/1.1

HTTP/1.1 404 Not Found
Content-Type: text/html
Date: Fri, 17 Jun 2016 08:13:29 GMT
Connection: Keep-Alive
Content-Length: 50591

top.md

top

The top command displays processor activity and also displays tasks managed by the kernel in real-time.

The following key strokes should be executed whilst top is running…

wireshark.md

wireshark

Wireshark is a network protocol analyzer.

It lets you see what’s happening on your network at a microscopic level.

To install (Mac OS X):

brew options wireshark           # check for options before installing
brew install wireshark --with-qt # install gui version
brew install tshark              # terminal oriented version of Wireshark designed for capturing and displaying packets without need for a gui

For CentOS:

yum install wireshark

With Docker:
--privileged required (otherwise: can't run /usr/sbin/dumpcap: Operation not permitted)
--security-opt seccomp:unconfined is an alternative

Once you have your pcap formatted file (see tcpdump) you can open Wireshark’s gui via your terminal by executing the shell command: wireshark

Once open you can use the gui to select “Open Capture File”, browse to your pcap file and select it.

Now you can start analysing your network traffic.

You can automate this process by reading in the capture file directly from the shell:

wireshark -r ~/Downloads/tcpdump-tests/0001.pcap

You can also specify the interface to connect to using -i (notice I had to use sudo in order to authorise Wireshark):

sudo wireshark -i en0

To see available interfaces execute: sudo wireshark -D (again you need sudo):

Capture-Message: Capture Interface List ...
Capture-Message: Loading External Capture Interface List ...
1. en0 (Wi-Fi)
2. awdl0
3. bridge0 (Thunderbolt Bridge)
4. en1 (Thunderbolt 1)
5. vboxnet1
6. en2 (Thunderbolt 2)
7. p2p0
8. lo0 (Loopback)

Every time there is (for example) a HTTP request, that might end up being 200 TCP packets, which is difficult to recognize and make sense of manually. But this can be simplified within Wireshark by clicking on Statistics -> Conversations, where it organizes all these disparate packets into TCP sessions.

tshark

When installing Wireshark you’ll also get a tshark command, which is a command line version of wireshark.

So you can read in your pcap file like so:

tshark -r ~/Downloads/tcpdump-tests/0001.pcap

This will display clearer formatted analysis than tcpdump -r provides.

tcpdump doesn’t know about HTTP or other network protocols. It knows pretty much everything about TCP but it doesn’t care what you put inside your TCP packets. tshark on the other hand knows all about what’s inside your TCP packets.

sudo tshark -i any \
            -R 'http.request.method == "GET"' \
            -T fields \
            -e http.request.method -e http.request.uri -e ip.dst

The above filters for just packets which have a HTTP GET request in them, and then prints out the request method and the URI for each one.

The way you filter results is by specifying -T and changing to the fields value. From there you can use the -e flag to specify how to filter data. So if you wanted to filter out all the DNS ttls from a tcpdump of just DNS traffic you could use something like:

tshark -r ~/dns-traffic.pcap -T fields -e dns.resp.ttl -e dns.resp.name

Note: if you open the pcap in wireshark, you can find the filter you need by selecting the data manually via the UI and then right-click’ing and selecting “Prepare a Filter > Selected”