forとかwhileの繰り返し処理をワンライナーで使う

職場の先輩がサクッと書いてたけど最初何してんのかわからなかった

こんな感じ。
切れ目がわからん

# zgrep "検索ワード" 対象ファイル | while read LINE ; do 変数1=`echo $LINE | awk -F':' '{print $1}'` ; 変数2=`echo $LINE | awk '{print $6}'` ; zgrep "$変数2" 変数1 | grep " from=" | awk '{print $1,$2,$3,$6,$7}' ; done | sort -n

# for 変数1 in `zcat 対象ファイル | grep "検索ワード1" | grep "検索ワード2" | grep "検索ワード3" | awk '{print $6}'` ; do zgrep $変数1 対象ファイル | grep "検索ワード4" | awk '{print $1,$2,$3,$6,$7}' ; done | sort -n

whileの方

解析

シェルスクリプトで使うあれっすよね。そこまでは分かったから。
分解してインデントつけるとみやすいかも。

while read LINE
do
    変数1=`echo $LINE | awk -F':' '{print $1}'`
    変数2=`echo $LINE | awk '{print $6}'`
    zgrep "$変数2" 変数1 | grep " from=" | awk '{print $1,$2,$3,$6,$7}'
done

1. 行が出力される間繰り返す
2. 処理開始
3. 変数1の定義　``内のコマンド出力結果なので、出力されたLINEの:区切り1カラム目が入る
4. 変数2の定義　``内のコマンド出力結果なので、出力されたLINEの半角スペース区切り6カラム目が入る
5. 変数2と1を引数に渡してzgrepとgrepを重ねる。結果はawkで1,2,3,6,7カラム目を出力
6. 処理終了

やってみた

メールアドレスとかあれなので、ssh attackのログでやってみる。
sshのログイン失敗のログは下記な感じで出力されている。
3行1setで一行目に失敗したユーザ名とIPが出力されてるので

# head /var/log/secure
～～略～～
Feb 12 06:00:37 tmin-server sshd[28342]: Invalid user test from 116.104.96.125
Feb 12 06:00:37 tmin-server sshd[28343]: input_userauth_request: invalid user test
Feb 12 06:00:37 tmin-server sshd[28343]: fatal: Read from socket failed: Connection reset by peer
Feb 12 06:00:40 tmin-server sshd[28344]: Invalid user admin from 116.104.96.125
Feb 12 06:00:40 tmin-server sshd[28345]: input_userauth_request: invalid user admin
Feb 12 06:00:40 tmin-server sshd[28345]: fatal: Read from socket failed: Connection reset by peer
Feb 12 06:00:43 tmin-server sshd[28346]: Invalid user pi from 116.104.96.125
Feb 12 06:00:43 tmin-server sshd[28347]: input_userauth_request: invalid user pi
Feb 12 06:00:43 tmin-server sshd[28347]: fatal: Read from socket failed: Connection reset by peer
Feb 12 06:00:44 tmin-server sshd[28348]: Invalid user admin from 116.104.96.125
～～略～～

こんな感じで抜き出せますね。

# grep -E 'Faild|Invalid' /var/log/secure*
～～略～～
/var/log/secure:Feb 12 06:00:37 tmin-server sshd[28342]: Invalid user test from 116.104.96.125
/var/log/secure:Feb 12 06:00:40 tmin-server sshd[28344]: Invalid user admin from 116.104.96.125
/var/log/secure:Feb 12 06:00:43 tmin-server sshd[28346]: Invalid user pi from 116.104.96.125
/var/log/secure:Feb 12 06:00:44 tmin-server sshd[28348]: Invalid user admin from 116.104.96.125
/var/log/secure:Feb 12 06:29:18 tmin-server sshd[28425]: Invalid user admin from 122.100.181.115
/var/log/secure:Feb 12 06:53:48 tmin-server sshd[28693]: Invalid user ubuntu from 195.50.5.226
/var/log/secure:Feb 12 06:53:55 tmin-server sshd[28695]: Invalid user oracle from 195.50.5.226
/var/log/secure:Feb 12 06:54:02 tmin-server sshd[28697]: Invalid user nagios from 195.50.5.226
/var/log/secure:Feb 12 06:54:10 tmin-server sshd[28699]: Invalid user git from 195.50.5.226
/var/log/secure:Feb 12 06:54:17 tmin-server sshd[28701]: Invalid user server from 195.50.5.226
/var/log/secure:Feb 12 06:54:25 tmin-server sshd[28703]: Invalid user dev from 195.50.5.226
/var/log/secure:Feb 12 06:54:32 tmin-server sshd[28705]: Invalid user oracle from 195.50.5.226
/var/log/secure:Feb 12 06:54:39 tmin-server sshd[28707]: Invalid user cs from 195.50.5.226
/var/log/secure:Feb 12 06:54:46 tmin-server sshd[28709]: Invalid user test from 195.50.5.226
/var/log/secure:Feb 12 06:54:54 tmin-server sshd[28711]: Invalid user vagrant from 195.50.5.226
/var/log/secure:Feb 12 06:55:01 tmin-server sshd[28713]: Invalid user vagrant from 195.50.5.226
/var/log/secure:Feb 12 06:55:08 tmin-server sshd[28718]: Invalid user vagrant from 195.50.5.226
/var/log/secure:Feb 12 06:55:16 tmin-server sshd[28720]: Invalid user developer from 195.50.5.226
/var/log/secure:Feb 12 06:55:23 tmin-server sshd[28722]: Invalid user developer from 195.50.5.226
/var/log/secure:Feb 12 06:55:30 tmin-server sshd[28727]: Invalid user developer from 195.50.5.226
～～略～～

:区切りの1カラム目はファイル名/var/log/secure
今回は変数2は半角スペース区切りの10カラム目、IPを入れてみましょう。

変数1にファイル名、変数2にIPが入るので、
上記grep結果の1行目

/var/log/secure:Feb 12 06:00:37 tmin-server sshd[28342]: Invalid user test from 116.104.96.125

の場合、実際の処理は

grep " 116.104.96.125" /var/log/secure | grep -E ' Failed|Invalid' | awk '{print $1,$2,$3,$7,$8,$10}'

となる。
該当ファイルでIPでgrepして、ユーザ名とIPが出力される行を抜き出して1,2,3,7,8,10カラム目を出力。
これをLINEが出力される限り繰り返し。

# grep -E 'Faild|Invalid' /var/log/secure* | while read LINE ; do FILENAME=`echo $LINE | awk -F':' '{print $1}'` ; IP=`echo $LINE | awk '{print $10}'` ; grep " $IP" $FILENAME | grep -E ' Failed|Invalid' | awk '{print $1,$2,$3,$7,$8,$10}' ; done | sort -n
～～略～～
Feb 8 06:47:39 user qm 173.192.101.210
Feb 8 06:47:39 user qm 173.192.101.210
Feb 8 06:47:39 user qm 173.192.101.210
Feb 8 06:47:39 user qm 173.192.101.210
Feb 8 06:47:39 user qm 173.192.101.210
Feb 8 06:47:39 user qm 173.192.101.210
Feb 8 06:47:39 user qm 173.192.101.210
～～略～～

たくさん出力されて見にくいのでちょっとsort+uniq
IPでsortしてユーザもIPも同じのはuniq
awkで出力調整。最後のカラムが回数。

# grep -E 'Faild|Invalid' /var/log/secure* | while read LINE ; do FILENAME=`echo $LINE | awk -F':' '{print $1}'` ; IP=`echo $LINE | awk '{print $10}'` ; grep " $IP" $FILENAME | grep -E ' Failed|Invalid' | awk '{print $1,$2,$3,$7,$8,$10}' ; done | sort -k6 | uniq -f4 -c | awk '{print $2,$3,$4,$5,$6,$7,$1}' | sort -n
Feb 1 00:32:41 user admin 86.96.65.236 1
Feb 1 04:22:35 user admin 185.153.19.85 1
Feb 1 05:41:20 user administrator 124.160.59.196 1
Feb 1 11:55:26 user admin 122.176.50.238 1
Feb 1 17:09:39 user admin 151.54.172.15 8
Feb 1 17:33:53 user ubnt 151.54.172.15 8
Feb 10 07:33:14 user bot1 5.189.168.214 173
Feb 10 07:33:16 user bot2 5.189.168.214 173
Feb 10 07:33:18 user arkserver 5.189.168.214 865
Feb 10 07:33:30 user arma 5.189.168.214 865
Feb 10 07:33:42 user bot 5.189.168.214 1211
Feb 10 07:33:58 user botmaster 5.189.168.214 1211
Feb 10 07:34:14 user bot 5.189.168.214 692
Feb 10 07:34:24 user butter 5.189.168.214 173
Feb 10 07:34:26 user csgo 5.189.168.214 1038
Feb 10 07:34:40 user csgoserver 5.189.168.214 1038
Feb 10 07:34:54 user csserver 5.189.168.214 173
Feb 10 07:34:56 user debian 5.189.168.214 173
Feb 10 07:34:58 user gmod 5.189.168.214 1038
Feb 10 07:35:12 user gmodserver 5.189.168.214 1211
Feb 10 07:35:29 user jenkins 5.189.168.214 346
～～略～～

ちなみにfindにgrep重ねてcutで取り出すとこれまた同じことができそう。

対象ファイル

# find /var/log -name "secure*"
/var/log/secure-20170129
/var/log/secure
/var/log/secure-20170122
/var/log/secure-20170212
/var/log/secure-20170205

xargsでcatに渡して失敗行出力

# find /var/log -name "secure*" | xargs cat | grep -E 'Faild|Invalid'
～～略～～
Jan 22 10:25:21 tmin-server sshd[11444]: Invalid user admin from 106.57.170.216
Jan 22 20:11:18 tmin-server sshd[13909]: Invalid user uwot from 82.85.187.101
Jan 23 00:32:00 tmin-server sshd[14771]: Invalid user admin from 45.76.134.144
Jan 23 04:24:07 tmin-server sshd[15628]: Invalid user odoo9 from 82.214.118.163
Jan 23 04:32:17 tmin-server sshd[15648]: Invalid user anonymous from 82.214.118.163
Jan 23 04:32:19 tmin-server sshd[15650]: Invalid user passwd from 82.214.118.163
Jan 23 05:25:13 tmin-server sshd[15763]: Invalid user admin from 59.49.25.107
Jan 23 06:10:18 tmin-server sshd[15872]: Invalid user mother from 58.19.145.22
Jan 23 09:37:51 tmin-server sshd[16555]: Invalid user support from 45.63.79.172
Jan 23 11:17:57 tmin-server sshd[16830]: Invalid user admin from 93.201.180.162
Jan 23 12:33:15 tmin-server sshd[17063]: Invalid user a from 178.207.153.61
Jan 23 12:33:23 tmin-server sshd[17067]: Invalid user pi from 178.207.153.61
Jan 23 12:33:25 tmin-server sshd[17069]: Invalid user osmc from 178.207.153.61
Jan 23 12:33:31 tmin-server sshd[17071]: Invalid user fmaster from 178.207.153.61
Jan 23 12:33:33 tmin-server sshd[17073]: Invalid user wildfly from 178.207.153.61
Jan 23 12:33:43 tmin-server sshd[17075]: Invalid user vnc from 178.207.153.61
Jan 23 15:16:42 tmin-server sshd[17741]: Invalid user admin from 121.12.42.113
Jan 23 17:11:58 tmin-server sshd[18441]: Invalid user user from 46.32.23.52
Jan 23 20:53:44 tmin-server sshd[19150]: Invalid user odoo9 from 210.44.144.31
Jan 23 20:53:45 tmin-server sshd[19152]: Invalid user odoo8 from 210.44.144.31
Jan 23 20:53:47 tmin-server sshd[19154]: Invalid user postgres from 210.44.144.31
Jan 23 20:53:49 tmin-server sshd[19156]: Invalid user odoo9 from 210.44.144.31
Jan 23 20:53:50 tmin-server sshd[19158]: Invalid user odoo8 from 210.44.144.31
Jan 23 20:53:51 tmin-server sshd[19160]: Invalid user odoo9 from 210.44.144.31
～～略～～

cut で区切りを半角スペースに指定して該当カラム抜き出し

# find /var/log -name "secure*" | xargs cat | grep -E 'Faild|Invalid' | cut -d " " -f 1,2,3,7,8,10
～～略～～
Jan 22 10:25:21 user admin 106.57.170.216
Jan 22 20:11:18 user uwot 82.85.187.101
Jan 23 00:32:00 user admin 45.76.134.144
Jan 23 04:24:07 user odoo9 82.214.118.163
Jan 23 04:32:17 user anonymous 82.214.118.163
Jan 23 04:32:19 user passwd 82.214.118.163
Jan 23 05:25:13 user admin 59.49.25.107
Jan 23 06:10:18 user mother 58.19.145.22
Jan 23 09:37:51 user support 45.63.79.172
Jan 23 11:17:57 user admin 93.201.180.162
Jan 23 12:33:15 user a 178.207.153.61
Jan 23 12:33:23 user pi 178.207.153.61
Jan 23 12:33:25 user osmc 178.207.153.61
Jan 23 12:33:31 user fmaster 178.207.153.61
Jan 23 12:33:33 user wildfly 178.207.153.61
Jan 23 12:33:43 user vnc 178.207.153.61
Jan 23 15:16:42 user admin 121.12.42.113
Jan 23 17:11:58 user user 46.32.23.52
Jan 23 20:53:44 user odoo9 210.44.144.31
～～略～～

sortしてuniqすると・・・・

# find /var/log -name "secure*" | xargs cat | grep -E 'Faild|Invalid' | cut -d " " -f 1,2,3,7,8,10 | sort -k6 | uniq -f4 -c

      1 Jan 31 23:55:20 user admin 1.160.122.233
      2 Feb 17 16:00:56 user admin 1.53.14.175
      1 Feb 17 16:00:57 user alex 1.53.14.175
      1 Feb 17 16:00:58 user test 1.53.14.175
      1 Feb 17 16:00:59 user pi 1.53.14.175
      1 Feb 17 16:01:00 user admin 1.53.14.175
      1 Feb 17 16:01:01 user ***** 1.53.14.175
      1 Feb 17 16:01:02 user guest 1.53.14.175
      1 Feb 17 16:01:05 user ubnt 1.53.14.175
      1 Feb 17 16:01:06 user PlcmSpIp 1.53.14.175
      1 Feb 17 16:01:07 user info 1.53.14.175
      1 Feb 17 16:01:09 user support 1.53.14.175
      1 Feb 17 16:01:10 user webmaster 1.53.14.175
      1 Feb 17 16:01:11 user git 1.53.14.175
      1 Feb 17 16:01:13 user user 1.53.14.175
      1 Feb 17 16:01:14 user ftpuser 1.53.14.175
      1 Feb 17 16:01:16 user Unknown 1.53.14.175

同じ感じ。1カラム目が回数になってるけど。

forの方

同じくちょっと分解

for 変数1 in `zcat 対象ファイル | grep "検索ワード1" | grep "検索ワード2" | grep "検索ワード3" | awk '{print $6}'`
do zgrep $変数1 対象ファイル | grep "検索ワード4" | awk '{print $1,$2,$3,$6,$7}'
done

1. 変数1に代入されるリストが``内のコマンド結果。対象ファイルのgrep抽出結果の6カラム目がリストになる
2. 変数1で対象ファイルをgrepして抜き出したいカラムをawkで切り出す。
3. 処理終了

whileと同じくssh attackのログで検証。
1. /var/log/secureの失敗ログの10カラム目を変数IPとしてリスト。
2. /var/log/secureのログからIPを抽出。userが出力される行でgrep。awkで切り出す。
3. 時間順sort

# for IP in `cat /var/log/secure | grep -E 'Faild|Invalid' | awk '{print $10}'` ; do grep $IP /var/log/secure | grep " user" | awk '{print $1,$2,$3,$7,$8,$10}' ; done | sort -n
Feb 12 06:00:37 user test 116.104.96.125
Feb 12 06:00:37 user test 116.104.96.125
Feb 12 06:00:37 user test 116.104.96.125
Feb 12 06:00:37 user test 116.104.96.125
Feb 12 06:00:40 user admin 116.104.96.125
Feb 12 06:00:40 user admin 116.104.96.125
Feb 12 06:00:40 user admin 116.104.96.125
Feb 12 06:00:40 user admin 116.104.96.125
Feb 12 06:00:43 user pi 116.104.96.125
Feb 12 06:00:43 user pi 116.104.96.125
Feb 12 06:00:43 user pi 116.104.96.125
Feb 12 06:00:43 user pi 116.104.96.125
Feb 12 06:00:44 user admin 116.104.96.125
Feb 12 06:00:44 user admin 116.104.96.125
Feb 12 06:00:44 user admin 116.104.96.125
Feb 12 06:00:44 user admin 116.104.96.125
Feb 12 06:29:18 user admin 122.100.181.115
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226
Feb 12 06:53:48 user ubuntu 195.50.5.226

IPでsortしてユーザ名とIPでuniq。→回数順でsort

# for IP in `cat /var/log/secure | grep -E 'Faild|Invalid' | awk '{print $10}'` ; do grep $IP /var/log/secure | grep " user" | awk '{print $1,$2,$3,$7,$8,$10}' ; done | sort -k6 | uniq -f4 -c | sort -n
      1 Feb 12 06:29:18 user admin 122.100.181.115
      1 Feb 12 07:46:38 user ubnt 166.122.78.220
      1 Feb 13 04:37:05 user admin 114.27.186.6
      1 Feb 13 12:35:56 user admin 74.66.229.110
      1 Feb 13 12:36:10 user oracle 61.148.167.42
      1 Feb 13 18:23:09 user admin 123.202.221.133
      1 Feb 13 19:52:22 user admin 78.194.25.54
      1 Feb 14 23:19:01 user admin 119.193.140.185
      1 Feb 15 22:16:33 user admin 211.223.82.164
      1 Feb 16 05:17:58 user admin 221.189.250.38
～～略～～
    556 Feb 13 22:34:57 user openerp 58.241.227.28
    556 Feb 13 22:35:22 user hadoop 58.241.227.28
    556 Feb 13 22:38:07 user usuario 58.241.227.28
    556 Feb 13 22:39:59 user backup 58.241.227.28
    556 Feb 13 22:40:17 user mosquitto 58.241.227.28
    556 Feb 13 22:44:18 user svn 58.241.227.28
    556 Feb 13 22:44:41 user cvs 58.241.227.28
    556 Feb 13 22:45:04 user info 58.241.227.28
    556 Feb 13 22:47:00 user nagios 58.241.227.28
    556 Feb 13 22:47:30 user weblogic 58.241.227.28
    695 Feb 13 22:33:32 user ubuntu 58.241.227.28
    695 Feb 13 22:34:08 user username 58.241.227.28
    695 Feb 13 22:37:02 user test 58.241.227.28
    695 Feb 13 22:40:40 user minecraft 58.241.227.28
    695 Feb 13 22:45:27 user vyatta 58.241.227.28
    695 Feb 13 22:45:57 user redmine 58.241.227.28
    695 Feb 13 22:46:28 user testuser 58.241.227.28
    834 Feb 13 22:32:54 user oracle 58.241.227.28
    973 Feb 13 22:48:13 user postgres 58.241.227.28
    973 Feb 13 22:49:33 user tomcat 58.241.227.28