Unix and Linux Commands

Last Update: 12/31/2019.

This is a collection of general *nix commands.

For commands of specific task (not found here), see the specific reference section of this website.

(For example, the commands about managing “file systems” are introduced at Linux File System — Docsnt documentation)

Prologue

Things are ordered from general fundamentals to specific tips.

Each category is a topic, a way to organize.

Philosophies and Notes

  • Everything is a “file”, including the shell, the devices, the directories.
  • Shell is very serious with spaces. Pay extra heed when writing scripts.
  • The brackets are meaningful in bash, I used them to represent user input if not specified otherwise.
  • Here is a small website for oneline commands.

The exit values of commands:

| Value | Status            |
| ----- | ----------------- |
| 0     | success           |
| non-0 | failure           |
| 2     | incorrect usage   |
| 127   | command not found |

Shell built-in vs external

  • some commands are only built-in in shell (man builtins)
  • some are only external (actually the majority)
  • others have both versions (e.g. kill)
    • one reason: avoid creating new process
help
command -V command
builtin kill
/bin/kill

Always read manual for the most authoritative help

man bash

Basics

  1. Open current path using GUI file explorer

    • Different systems have different file explorers, it’s hard for me to test each one
    • See here for more
    • Pay attention to the privileges, it’s generally not good to run as root
    nautilus . # Ubuntu?
nemo .
dolphin . # Arch Linux?

  2. Redirection and pipes

    • Sometimes you want to save output of a command as a file
    • Or you want to use a file as input of a command
    • Or you want to use output of a command as input of another file
    • different streams (FD):
      • 0: stdin
      • 1: stdout, default for > and >>
      • 2: stderr
    • /dev/null is a magic file, everything redirected to it is magically discarded
    • order of redirection does not matter
    # a.txt will be cleared and only consist of this line
echo "write this line into file a.txt" > a.txt
# the original content of a.txt would not be affected
echo "append this line into file a.txt" >> a.txt

# feed a small c program that reads input from STDIN with test_input
./a.out < test_input
# store the errors and discard the standard output
./a.out 2>> err_log 1> /dev/null
# two streams go to the same destination (2 is tied with 1)
./a.out > afile 2>&1
# order does not matter
> count.txt 2> log.txt wc < shakespeare.txt

# only output lines containing "file"
man cat | grep file
# list all files in root directory and output only the last 10 lines
ls -lha / | tail -n 10

cat << TerminationSign
> asdfdas
> asdfdsa
> TerminationSign (can be anything)

cat << END > ShortScript.sh
> #!/bin/bash
> echo "I love bash"
> END

  3. Escape and back quote

    • to escape bash keywords, \
    • to use the output of a command, shell will treat strings inside `` as commands
    echo -e "bash\tis\tgood!"
echo `echo "I like bash!"`

  4. Test and true and false

    • true or false
    • ‘[’ is the same as test
    test
true
false

  5. Change shell mode

    set -o vi # set vi mode

  6. Multiple lines

    • use ‘;’ to run multiple commands in one line
    • use ‘\’ to separate one-line commands
      • note that you will see the so-called second prompt (PS2) here, usually “>”
    echo this; echo that
echo "this \
that"

  7. Comments

    #

  8. Quote

    "" # weak quote
   # $var are variables
   # back quotes and escape characters are processed
'' # strong quote, everything is string
`` # back quote, the primary output of the command inside with replace this string

  9. Aliases

    • alias is just shortcuts or macros
    • use \ at the beginning of a command to un-alias
    alias "rm -rf"="echo 'rm? how dare you?'"
\rm -rf ./*.class

  10. Expansions

    echo $( date ) # substitute with the command output
               # spawn a new process, that process write to stdout, current process reads from that stdout

mkdir -v ~/{old,new} # creates ~/old, ~/new
echo {1..10}

echo ~user # home directory of 'user'
echo ~+ # $PWD

echo $(( 1+1 )) # https://www.gnu.org/software/bash/manual/html_node/Shell-Arithmetic.html

str=1234abcd
echo ${str:1:2} # substring
echo ${str: -1:2} # negative offset must have a space to avoid confusion with ${parameter:-word}

# case conversion (pattern should be matching one char only)
str=aabbccdd
echo ${str^a}  # convert the first char, if matched with pattern, to upper case
echo ${str^^a} # convert all
str=AABBCCDD
echo ${str,a}  # lower case
echo ${str,,a}

set -- a b c d e f g 1 2 3 4 # set the positional parameters, $1 $2 etc.
array=(1 2 3 4 a b c d e f g)

echo ${#str} # length of string
echo ${#@}
echo ${#*} # length of positional parameters
echo ${#array[@]}
echo ${#array[*]} # length of array
# ... read more https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html

# process substitution
echo <(echo "bash love")
echo < <(echo "bash love")
echo "bash love" > >(cat)

Keyboard Shortcuts

^ stands for Ctrl

  1. Abort (SIGINT)

    ^c

  2. Suspend (signal 17, SIGTSTP)

    ^z

  3. Go to the beginning of line

    ^a

  4. Go to the end of line

    ^e

  5. Delete content after cursor

    ^k

  6. Clear the screen

    ^l

  7. Log out current session

    ^d

  8. Search in the history

    ^r

  9. Autocompletion of commands or paths (Press twice to see all available ones if not unique)

    • maybe it’s required to type a letter first then press Tab twice?
    [Tab]

  10. Go to last command

    [Up Arrow]

  11. Fetch the parameters of last command

    • Press together
    [Esc]+.

  12. Pause/Freeze the terminal

    ^s

  13. Resume the terminal

    ^q

For cursor movement, here is an excellent picture.

Files

  1. Create blank file or update last-change time

    • -r/–reference=FILE use another file’s time as reference
    • -a only change the access time
    • -m only the modification time
    • -c/–no-create do not create new file
    touch                          # create an empty file or update access and modification time
touch -t [[CC]YY]MMDDhhmm[.ss] # change both access and modification time to specified time

  2. See file type and char-set and other information

    file
file -i [file]

  3. See file disk usage

    • -h human-reading friendly
    • -s sum
    • -a all files not just directories
    • -dN max-depth=N
    du
du -sh
du -s *
du -hs .[^.]*
du -sch .[!.]* * | sort -h
du -hd1 | sort -h
du -ahd1 | sort -h

  4. Copy

    • -r dir
    • -v verbose
    • -p reserve time
    cp [source] [target]
scp [source] [target] # secure copy, copy across ssh and so on

  5. rsync

    • -n dry-run
    • -a archive
    • -v verbose
    • -h human-friendly size
    • -u update, skip newer files on the receiving side
    • remove dry-run after checking the result
    rsync -avh --dry-run <source> <target>    # copy and overwrite, no deletion
rsync -u -ah --dry-run <source> <target> # copy and overwrite for files that are newer in source

  6. Move and rename

    • -v verbose
    • mv treats directories and files the same
    mv [source] [target]

  7. Delete

    • -i interactive
    • -r dir
    • -f force
    • rm -rf ./* delete all files in current dir
    • su”do r”m -r”f /* Say goodbye to your system (Never run this!!!!!)
    rm

  8. Quota

    quota -v

  9. File system disk space usage

    • troubleshoot
    df
df .
df -k  # use 1k as SIZE scale

Pay heed to the difference between du and df

Directory

  1. Create new dir

    • -p Create parents if necessary
    • -v verbose
    mkdir
mkdir -pv [name]

  2. Remove empty dir

    • Note: use “rm -r” to remove non-empty dir
    rmdir
rmdir -pv

Find

  1. Find file by name using prebuilt databases

    • Like pre-processing the files
    • the database is updated per day by default
    • cannot lookup the file changes after the last update
    locate [pattern-to-find]
updatedb # update the database, may be slow

  2. Find

    • Advanced find
    • Specify the path
    • More options
    • Real-time scan
    • path
      • . current directory
      • / root directory (all files)
    • arguments
      • -name [name]
      • -perm [permissions]
        • can only use octal representation
      • -type
        • d directory
        • l link
      • -user
      • -group
      • -ctime change time
      • -size
    • use the result to execute commands
      • -exec
    find [path] [arguments]

# call ls -l on all files in current directory whose names start with 'a'
find . -name "a*" -exec ls -l {} \;

# search all files in current directory for pattern
# and print the paths of those files
find . -type f -print0 | xargs -0 grep "pattern"
# only files, no paths
find . -type f -print0 | xargs -0 grep -l "pattern"

  3. Where is this command/program I’m executing?

    which
which which

Archive and Compress

How are zlib, gzip and zip related? What do they have in common and how are they different?

  1. Zip and Unzip

    zip [archive-file-name] [files-to-archive]
zip a.zip a

unzip file[.zip]

  2. gzip and gunzip

    • compress
    gzip
gunzip

  3. tar

    • archive only, no compress
    • -c create
    • -x extract
    • -v verbose, can (should?) omit when there is a number of files
    • -z gzip
    • -f use archive file or device ARCHIVE
    # archive a file
tar -cvf [archive-file-name] [files-to-archive]
tar -cvf a.tar a.b b.c
tar -chof # h: follow symlinks o: Compatibility

# archive and compress
tar -cvzf [archive-file-name] [files-to-archive-and-compress]
tar a.tar.gz a.b b.c

# extract a archive
tar -xvf [archive-file]
tar -xvf a.tar

# extract and uncompress
tar -xvzf [compressed-archive-file name]
tar -xvzf a.tar.gz

# extract to different directory
# the target director must exist
tar -xvzf a.tar.gz -C [/target/directory]

Help

  1. help

    • Almost all commands have help option
    • It’s a summary of the usage
    • This is always the first option before using an unfamiliar command
    [command] -h
[command] --help

help [command] # a command called help, mainly for shell build-in commands

  2. man (manual)

    • Explain a command
    • If exists, more detailed than help
    • A detailed usage instruction
    • You can even look up C language libraries
    • Types (-s<TYPE_NUM>) (from man man)
      1. Executable programs or shell commands
      2. System calls (functions provided by the kernel)
      3. Library calls (functions within program libraries)
      4. Special files (usually found in /dev)
      5. File formats and conventions eg /etc/passwd
      6. Games
      7. Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
      8. System administration commands (usually only for root)
      9. Kernel routines [Non standard]
    • Ex: use -s3 to check section 3 specifically
      • otherwise will lookup in the pre-defined order: “1 n l 8 3 2 3posix 3pm 3perl 5 4 9 6 7”
      • or change here: /etc/manpath.config
    • -k [key word]
      • search for all documentation with key word
    man ascii
man -s3 malloc # c function, i.e. the function within program lib
man echo

  3. info

    • more detailed than man

  4. doc

    • txt, html, pdf
    • stored at /usr/share/doc

  5. What is this command (one-line description)

    whatis

  6. Forget how to spell a command

    apropos

Previous Commands

  1. See history

    history

The event reference (!) is mainly used in scripts?

  1. Run the last command

    !!

  2. Run the last command beginning with a specific letter

    ![specific letter]

  3. Run the command by number in history (use history to see the number)

    ![number]

  4. Execute the command containing specific string

    ![specific_string]

  5. Execute the nth preceding event (i.e. using a negative number for the index)

    !-[number]

  6. Last word of previous command

    • Can be used to test if in same shell
    • Safe way to rm
    !$
ls xxx.yyy
rm !$

  7. Replace part of previous commands

    • I think only few people know this
    • sample command from @bashoneliners
    # change encoding of all files in a directory and subdirectories
find . -type f  -name '*.java' -exec sh -c 'iconv -f cp1252 -t utf-8 "$1" > converted && mv converted "$1"' -- {} \;
^java^cpp

# safe way to remove
ls xxx
^ls^ rm -v

Input and Output

  1. Display a line of text

    • -n no new line at the end
    • -e expand special characters and variables
    echo
echo "xxx" >> [file]
echo "xxx" > [file]
echo -e "$!"

  2. Display all the content of one file

    cat
cat a.txt > b.txt

  3. Display the head of one file

    • -n Display the first n lines
    head
head -n 10

  4. Display the tail of one file

    • -n
    • -f Follow the latest updates of one file (usually used to watch the log files)
    • -F (recommended) same as -f but “keep trying to open a file if it is inaccessible”
      • can track when renaming
      • avoid confusion when logs are rotated
    tail

  5. Paging through text (downward only)

    • q to quit
    more

  6. Paging through text

    • q to quit
    • less is more
    less

  7. Read input

    read var_1
# type input
echo "$var_1"

  8. Watch a command, namely keep printing the output of a command

    • -n interval
    • -c color
    watch -c -n 1 command

  9. Copy stdin to multiple places

    • -a –append append not overwrite
    • -i –ignore-interrupts signals
    • –output-error[=mode] change the error behavior, see man page
    tee

echo "some-huge-file" \
  | tee some-huge-file.file | sha1sum > some-huge-file.sha1

tar -cvf - "a-file" \
  | tee >(md5sum --tag) >(sha256sum --tag) > "a-file.tar"

  10. Print printable strings

    strings [some-file]

Filters and Text Manipulation

  1. Search for specific pattern

    grep [file] [string]
pdfgrep # grep in pdf files

  2. sort

    • sort based on character or number
    • use alphabet and numbers by default
    • -r reverse
    • -n sort numerically
    • -f ignore case
    • -u unique, remove duplicated lines
    • -t SEP use SEP as delimiter/separator
    • -k KEYDEF specify the key to use when sorting
    ps -aef | sort > out.txt

  3. Remove consecutive identical lines

    • Usually needs to be sorted first
    • -c precede line with number of repetitions
    • -d only repeated lines
    • -u only non-repeated lines
    uniq

  4. Word count

    wc
wc -l # lines
wc -w # words
wc -c # bytes
wc -m # characters

  5. diff

    • output means the edits that is required to make file1 identical to file2
      • append
      • delete
      • change
    • ‘<’ means line from file1
    • ‘>’ means line from file2
    diff file1 file2
vimdiff

  6. Cut text

    • Used to process specific columns of the delimited text
    • -d specify the delimiter, TAB by default
    • -f specify the number of column to output
    • -c cut based on characters
    cut
cut -b [bytes] # select only specified bytes
cut -c 1-2, 3, 6 # characters
cut -d ' ' # use ' ' as delimiter

cut -d: -f1 /etc/passwd
grep some-user /etc/passwd | cut -d: -f2,3,4
cut -c2-6 /etc/passwd # output from 2nd character to 6th character

  7. Paste

    • Merge lines of files
    • like reverse of cut
    paste
paste -d ' ' file1 file2 file3 > output

  8. Check spelling

    aspell
aspell check <some-file>
aspell list < <some-file>

  9. Translate

    • needs redirection
    tr -d 'some-words' < <some-file> # delete some-words from some-file
tr 'a-z' 'A-Z' < <some-file> # change case

  10. Search and replace

    • regular expression
    • write to stdout
    sed 's/aaa/bbb/g' <some-file> # replace 'aaa' with 'bbb', 'g' for global
sed '1,50s/aaa/bbb/g' <some-file> # specify the lines
sed -e 's/aaaa/bbb/g' -e 's/ccc/ddd/g' <some-file>
sed -f <some-script> <some-file> # use a file as script

Date and Time

  1. Show or set current date and time

    • +%Y–%m–%d (e.g.)
      • Print time in specific format
      • % is the identifer
    • -u Greenwich time
    • -s setting
    date

  2. Show hardware time (time on the motherboard)

    hwclock
clock

  3. Calendar

    cal

  4. Up time

    • troubleshooting
    • Output is consist with:
      • uptime
      • Login users
      • load
        • average number of processes in “run” or “ready” queue of 1 min, 5 min and 15 min
        • note that machine may have more than one CPU
        • 1 means 100% load for one CPU
        • not that fantastic anymore since majority of systems run on multi-core CPUs
      • also consider processes blocked due to I/O, shown as load
    uptime

Names

  1. System

    uname
uname -a

  2. Host

    hostname

Users

  1. Switch user

    • if empty, switch to root
    su - [user]

  2. Switch user and keep current path

    • if empty, switch to root
    su [user]

  3. Run as root

    sudo
sudo -E                # pass environment
sudo SOME_VAR=SOME_VAL # pass variable

sudo -i # a shell

[sudo] SOME_CMD | sudo tee SOME_WRITE_FILE    # redirection
[sudo] SOME_CMD | sudo tee -a SOME_WRITE_FILE # redirection append

  4. Log out

    exit

  5. Change password

    passwd

  6. User Id

    id
id 1000 # display info about user whose id is 1000

  7. Who am I?

    • The informal philosophy of commands: the shorter, the more information; the longer, the more specific.
    whoami
who
w

  8. Create and delete user

    # add
useradd
adduser # a wrapper of useradd
# passwd [user] # set password if not already

# del
deluser --remove-home [user] # Debian-based
userdel -r [user]            # RHEL-based

  9. Manage group

    groupadd
addgroup # a wrapper of groupadd

groupdel
delgroup # a wrapper of groupdel

newgrp [group]   # change current group ID
newgrp - [group] # with reinitialized environment
newgrp           # change to default group in /etc/passwd file

  10. Fingers

    finger [user] # display user's information if exists
chfn # change finger

  11. Administer commands

    gpasswd # administer /etc/group and /etc/gshadow
yppasswd # ?
ypchfn # ?

  12. Log out old sessions that was ‘broken’

    pkill -u [username]

  13. Last logged in user

    • -a: display hostname in last column
    • -i: display ip
    last -5
last -5ai

Privileges and Permissions

ls -a can list the permissions of a specific file.

  1. Change mode

    • The permissions in *nix is in U/G/O model
    • The representations of permission is ordered by user/group/other
    • There are 3 permissions of a file:
      • r: read, 2^2=4
      • w: write, 2^1=2
      • x: execute, 2^0=1
    • As you can see, the largest permission represented by an integer is the sum of them, 7
    • Try not messing up with permissions until mastering them
    • -R recursive change permission of all files in a directory
    • -f force
    • [ugo][+-=][rwx]
    • Unix Permissions Calculator
    • See other places or man for details
    • Note that the directories must have permission execute ‘opened’ to be opened
      • r for directories: can read the list
      • w for directories: create or remove files
      • x for directories: can pass through
    • There are other special permissions. For example, ‘s’ is the setuid bit, meaning that the user who issued the command did not change (no sudo) but the user receives the septics privileges, like passwd and ping.
    chmod -R [directory]
chmod -f ...
chmod u+x [file]
chmod ug-x [file]
chmod 640 [file]
chmod u+x,g-x [file]
chmod u=rwx [file]

  2. Change ownership

    • -R: recursively change all subdirectories and files
    chown some-user some-file
chown -R some-user some-directory

  3. SELinux (advanced way to manage permissions)

    sestatus
getenforce
seinfo
semanage
setenforce 0 # put SELinux in permissive mode (Not Recommended)
             #   SELinux policies will not take effects
             #   but will log
             #   and label newly created files
             # will reset after rebooting

# modify one of the following files to disable
/etc/grub.conf
/etc/selinux/config

  4. Default permission when creating a new file

    • It’s negated (reversed)
    • details
    umask

Process Management and Resource Monitor

  1. Run in background

    [command]&

  2. See processes in background

    • Including the commands executed with & at the end
    • and the commands suspended using Ctrl+Z
    jobs
jobs -l

  3. Put a process in background if not started with &, or resume suspended processes (SIGTSTP) in background

    bg [ID]

  4. Put a process back to foreground

    • note the output may still be absent
    fg [ID]

  5. Modify scheduling priority of a program

    • -20 is most favorable
    • 19 is least favorable
    nice -19 [command]

  6. List processes

    • troubleshoot
    • See IDs of processes
    • Read help of ps for details
    • e: all processes
    • a: all processes except session leader and processes not associated with a terminal
    • x: display session leader and processes not associated with a terminal (?)
      • e = a + x ?
    • u: userlist
    • w: wide, use twice for unlimited width
    ps
ps ef   # all processes
ps aux  # no parent process ID

ps -ef | grep mysql | grep -v grep | awk '{print $2}' # lookup PID of mysql

ps axuf
ps auxww           # output with unlimited width

ps aux | grep -v grep | grep -i -e VSZ -e
ps -e -o pid,vsz,comm= | sort -n -k 2

ps -C bash -o pid= # show pid of bash
pidof bash         # show pid of bash

  7. Monitor processes

    pstree -a
top
htop
slabtop

  8. Send signals to process

    • List of “sigspec” and “signum”:
      • SIGHUP 1
      • SIGINT 2
      • SIGQUIT 3
      • SIGFPE 8
      • SIGKILL 9
      • SIGALRM 14
      • SIGTERM 15
    • Use kill -l for more
    • process can ignore some signals
    kill -l

# send SIGTERM by default
kill [id]
kill -9 [id] # the kill signal cannot be ignored and immediately
             # skipping the "clean up" before dying

kill -0 [id] # check if process exists or permission to signal exists

# note that built-in kill is different from the external kill
# http://man7.org/linux/man-pages/man2/kill.2.html
kill -s [sig] 0  # send sig to every process in the process group of the calling process
kill -s [sig] -1 # all processes except the kill process itself and init
# If pid is less than -1, then sig is sent to every process in the process group whose ID is -pid.

  9. Run commands that ignore some signals

    trap [command list] [signal]

  10. Do not kill command when log out

    nohup [command] > /dev/null 2>&1 &

  11. Show available memories

    • troubleshoot
    free
free -h
free -mt

  12. Read syslog

    • troubleshoot
    sudo tail -F /var/log/syslog

  13. system control

    sudo systemctl status nginx
sudo systemctl list-unit-files

  14. Journals / logs

    • troubleshoot
    sudo journalctl -u docker -b # unit boot
sudo journalctl -u docker -f # follow
sudo journalctl -u docker --no-pager # print to stdout
sudo journalctl -u docker --no-pager > ./some-log # and redirect to a file

  15. Virtual Memory Stat

    • troubleshoot
    • r: running
    • b: blocked
    • io
      • bi: Blocks In
      • bo: Blocks Out
    • in: interrupts
    • cs: context switches
    • cpu (in %)
      • USer
      • SYstem
      • IDle
      • Wait
    • can leave it run for some time and see the trends
    vmstat
vmstat 1 # per second, output is 1 second average

  16. IO Status

    • troubleshoot
    iostat
iostat 1 # per 1 sec
iostat -x

  17. Trace Syscalls

    • troubleshoot
    strace [some-command]
strace -e [some-key-word] [some-command]

Network

  1. Net Status

    • troubleshoot
    netstat -s          # show a sweet summary
sudo netstat -lnp   # show all open ports, including sockets
sudo netstat -tupln # without sockets

netstat -an    # all and numerical
netstat -tan   # tcp
netstat -tanep # extended and show program

  2. Investigate sockets

    sudo ss -lnp

  3. List activating ports

    lsof

  4. Firewall / iptables

    sudo iptables -Lnv
sudo ip6tables -Lnv
sudo dpkg-reconfigure iptables-persistent # iptablessave for debian

# delete all rules
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo ip6tables -P INPUT ACCEPT
sudo ip6tables -P FORWARD ACCEPT
sudo ip6tables -P OUTPUT ACCEPT
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -F
sudo iptables -X
sudo ip6tables -t nat -F
sudo ip6tables -t mangle -F
sudo ip6tables -F
sudo ip6tables -X

  5. IP, interfaces, routes etc.

    • check man ip
    ip
ip route
ip link show

  6. Enable bbr

    echo "tcp_bbr" >> /etc/modules-load.d/modules.conf
echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf
echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf
sudo sysctl -p

# test
sudo sysctl net.ipv4.tcp_available_congestion_control
sudo sysctl net.ipv4.tcp_congestion_control

  7. Curl

    curl --help
curl --manual

# GET
curl [URL]

# POST
curl -d data=data [URL]
curl --data data=data [URL]
curl -d '{"json":"value"}' -H "Content-Type: application/json" [URL]
curl --data-urlencode [data] [URL] # auto encode url for POST
curl --form upload=@[file_name] --form press=[some_value] [URL] # RFC1867-posting upload file

# HEAD
curl --head [URL]
curl -I [URL]

# PUT
curl --upload-file [some_file] [URL]

# Get more details
curl -i # include response headers
curl --verbose
curl -v # abbr for verbose
# record everything sends and receives
curl --trace [dump_file_name]
curl --trace-ascii [dump_file_name]
curl --trace-ascii [dump_file_name] --trace-time

# output
curl -o
curl -O

# multiple urls
curl [URL1] [URL2] # send same request to multiple urls
curl [URL1] --next -I [URL2] # send different requests to multiple urls

# referer
curl --referer [origin_url] [destination_url]

# user agent
curl --user-agent "user agent string" [URL]

# follow redirection
curl --location [URL]

# Cookie
curl -b # abbr for --cookie
curl --cookie "data=data" [URL] # send cookie for GET
curl --dump-header [cookie-dump-file] [URL] # record the cookies
curl --cookie [previous-dumped-cookie] [URL] # user previous stored cookies
# preferred way to store cookies, use non-exist file as --cookie parameter to enable cookie engine only
curl --cookie [previous-cookie-file-name] --cookie-jar [new-cookie-file-name] [URL] # use old cookie and store new

# https
curl -k # abbr for --insecure
curl --insecure # not verifying server-side certificates
curl --cery some-cert-file.pem [https_URL] # use client-side certificate
curl --cacert ca-bundle.pem [https_URL] # use CA store to verify server-side certificates

# specify a ip address for a domain
curl --resolve [some_domain]:[some_port]:[some_address] [same_domain]

curl --proxy [some_proxy] [URL]

  8. wget

    • directly download a link
    • q: quite
    • O: output, - for stdout
    • P: prefix directory (will create sub-directory if non-exist)
    wget 127.0.0.1/file.txt
wget -qO- 127.0.0.1
wget -P ~/download/new-directory 127.0.0.1/files.txt

  9. Actual Networking Traffic

    • troubleshoot
    tail -F /proc/net/dev
iftop
iptraf
tcpdump # raw

Free Cache

How to Clear RAM Memory Cache, Buffer and Swap Space on Linux

How do you empty the buffers and cache on a Linux system?

  1. Use slabtop display kernel slab cache information:

    slabtop
vmstat -m
cat /proc/slabinfo

  2. Drop buffer to free memory

    sync

  3. Drop cache

    # as root usuer
## To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
## To free dentries and inodes
echo 2 > /proc/sys/vm/drop_caches
## To free pagecache, dentries and inodes
echo 3 > /proc/sys/vm/drop_caches

# using sudo
sudo sh -c 'echo 1 >/proc/sys/vm/drop_caches'
sudo sh -c 'echo 2 >/proc/sys/vm/drop_caches'
sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
## or
echo "echo 1 > /proc/sys/vm/drop_caches" | sudo sh

Environment and Global Variables

Special Parameters

Parameter and Variable Index

  1. Get env/configs for system

    printenv
printenv | less
set
set | less
getconf ARG_MAX

set path = ($path ~/exe/bin)

  2. Directly print current variables

    echo $? # return value of last command
echo "program name is: $0"
echo "the first command line parameter is: $1"
echo "there are $# command line parameters"
echo "$@" # an array of command line parameters
echo "$*" # a list of command line parameters

echo "$$" # current process ID, if in a shell then shell's, if in a script then the process that is running the script
echo "$BASHPID" # process ID of the current instance of bash
echo "$BASH_SUBSHELL" # "subshell level", it's a variable
echo "$!" # the process ID of the most recently executed background pipeline

echo "$OSTYPE"
echo "$USER" # your login name
echo "$HOME" # the path name of your home directory
echo "$HOST" # the name of the computer you are using
echo "$ARCH" # the architecture of the computers processor
echo "$DISPLAY" # the name of the computer screen to display X windows
echo "$PRINTER" # the default printer to send print jobs
echo "PATH" # the directories the shell should search to find a command

Scripting

Shell scripts need a shebang to be run directly, specifying its “caller” program

#!/usr/bin/env bash

echo "this is a sample script file"

This is same as

/usr/bin/env bash [the-script-name]

  1. Basic scripting

    echo "program name is: $0"
echo "the first command line parameter is: $1"
echo "there are $# command line parameters"

expr 2 + 3
echo $((2**3))
b=`expr $a+1` # no extra spaces

array=(1 2 3)
echo ${a[*]}; echo ${a[0]}

x=3
echo $x
unset x # delete x
echo $x

mydir=`pwd`; echo $mydir

echo $1
echo "$1" # In general, use double quotes in case the variable is not defined

  2. Write script on the fly

    • capture all commands and save to a file
    • Ctrl+D (EOF) to end
    script

  3. Expression and calculation

    • expr commands supports:
      • arithmetic operators: +,-,*,/,%
      • comparison operators: <, <=, ==, !=, >=, >
      • boolean/logical operators: &&, ||
      • parentheses: (, )
      • precedence is the same as in C, Java
    • bc is a precision calculating language
    • awk can also be used to evaluate expressions
    echo $((1+3))
if [ `expr 1 < 2` ]; then echo "yeah"; fi
echo "scale=6;(1/12)+(7/13)" | bc
echo "(1/12)+(7/13)" | bc -l
awk "BEGIN {print -3.14 - -1.4141 * 3}"

  4. Test

    [[ string1 = string2 ]] # True if strings are identical, note only ONE equal sign
[[ string1 == string2 ]] # Same as above
[[ string1 != string2 ]] # True if strings are not identical
[[ string ]] # Return 0 exit status (=true) if string is not null
[[ -n string ]] # Return 0 exit status (=true) if string is not null
[[ -z string ]] # Return 0 exit status (=true) if string is null

[[ int1 –eq int2 ]] # Test identity
[[ int1 –ne int2 ]] # Test inequality
[[ int1 –lt int2 ]] # Less than
[[ int1 –gt int2 ]] # Greater than
[[ int1 –le int2 ]] # Less than or equal
[[ int1 –ge int2 ]] # Greater than or equal

test 1024 -eq 1024 # can use test directly

if cond1 && cond2 || cond3
if cond1 –a cond22 –o cond3
if [ "$a" –lt 0 –o "$a" –gt 100 ]

test -d file # if file is a directory
test -f file # if file is not a directory
test -s file # if the file has non zero length
test -r file # if the file is readable
test -w file # if the file is writable
test -x file # if the file is executable
test -o file # if the file is owned by the user
test -e file # if the file exists
test -z file # if the file has zero length

  5. Flow controls

    if []; then
    echo "1"
elif []; then
    echo "2"
else
    echo "3"
fi

for i in 1 2 3 4
do
    echo i
done

while test "$i" -gt 0
do
    i=`expr $i - 1`
done

until test "$i" -lt 0
do
    i=`expr $i - 1`
done

status="off"
input="on"
case $argument in
"on") echo "on"
    status="on";;
"off")
    status="off"
    echo 3
    ;;
esac

  6. Substring Operations

    • Strip variables
    • Replacement
    statement="Bashhhhhh? BashBaassshhh!"
echo "${statement#B*h}"   # shortest from beginning
                          # hhhhh? BashBaassshhh!
echo "${statement##B*h}"  # longest  from beginning
                          # !
echo "${statement%B*\!}"  # shortest from ending
                          # Bashhhhhh? Bash
echo "${statement%%B*\!}" # longest  from ending
                          # (empty string)

mydir=$(pwd)
basename $mydir # only the last directory name
dirname $mydir  # no `basename`

# replacement
echo "${statement/hhh/HHH}"  # replace first occurrence
                             # BasHHHhhh? BashBaassshhh!
echo "${statement/#Bash/HHH}"  # replace front-end first occurrence
echo "${statement/%hhh\!/HHH}" # replace back-end first occurrence
echo "${statement//hhh/HHH}" # replace all occurrences
                             # BasHHHHHH? BashBaasssHHH!
echo "${statement//hhh/}"    # remove all occurrences (replace with empty string)
                             # Bas? BashBaasss!

  7. Positional parameters

    • either passed as command line args, from function calls, or set
    • accessed via $0, $1, …
      • $0 is the name of current script/function
    • $# number of positional parameters or cmd line args
    • “$*” must be quoted, all positional parameters as a single word
    • “$@” should be quoted, all positional parameters as separated words, each as a quoted string without interpretation or expansion
    • use shift to move them, i.e. $1 will hold the value of $2, $2 will be $3 and so forth
      • $1 will be lost, $0 will not change
    echo "$2"
shift
echo "$1" # same thing

# read command line arguments
while [[ $# > 0 ]];do
    key="$1"
    case $key in
        -n|--name)
        NAME="${2}"
        shift # we read 2 args
        ;;
        -h|--help)
        HELP="1"
        ;;
        -f|--force)
        FORCE="1"
        ;;
        *)
        # unknown
        echo "Invalid arg detected, aborting..."
        exit -1
        ;;
    esac
    shift
done

  8. Exec

    # stdin
exec 6<&0          # Link file descriptor #6 with stdin. Saves stdin.
exec < input-file  # stdin replaced by file "input-file"
read var           # read first line in input-file to var
exec 0<&6 6<&-     # restore stdin from file descriptor #6 and free #6
exec <&6 6<&-      # restore stdin from file descriptor #6 and free #6

# stdout
exec 7>&1           # Link file descriptor #6 with stdout. Saves stdout.
exec > output-file  # stdout replaced by file "output-file".
echo "Editing file"
exec 1>&7 7>&-      # Restore stdout and close file descriptor #7.

# no extra process
exec bash -l
ps                  # new bash will have same process ID as current shell

bash -l             # create a new bash shell
ps                  # will have (at least) 2 shell processes

# subshell problem (inaccessible variables within a subshell)
num=0
cat input | read num  # first line of input is a non-zero number
echo $num             # num is still 0

# exec for subshell problem
num=0
exec 3<> input        # set file descriptor #3 to input, create if not exist
read num <&3
echo $num             # num is the number in input file

  9. eval

    • combine strings into a single command and execute
    • useful if args are only known at run-time
    help eval # read the usage help for your own shell
eval echo -e $USER

  10. array

    • note that bash variables have no type
      • strings
    • Arrays
    declare -a array # declare array as an array, not necessary in all cases

# print whole array
echo ${array[@]}
echo ${array[*]}
# difference is same as $@ vs $*

# normal variables can use array operators
str=123
echo ${#str[@]} # length is 1
echo ${str[0]} # will be 123

# literal init
array=( {0..3} AAA )
echo ${array[@]}
array[10]="This is the 10th elem of a sparse array"
echo ${array[@]}
echo ${!array[@]}

# another init
array=([1]=one [2]=two)
echo ${array[@]}

# read to array
read -a inputs

# can use variable as index
idx=1
echo ${array[idx]}

# same as ${array[0]}
echo ${array}

# for-each
for elem in "${inputs[@]}"
do
    echo $elem
done

# can use stirng operations, e.g. substring removal (strip) and replacement etc.
echo ${array[@]#o*}
echo ${array}

# delete
unset array[0]
unset array

Schedule

  1. Chronos

    crontab -e
(crontab -l 2>/dev/null; echo "*/5 * * * * <some-job>") | crontab -

  2. Chronons Job

    Minute Hour DayOfMonth MonthOfYear DayOfWeek Year Command

* * * * * *          each minute
59 23 */2 * *        each 2 day at 23:59
@reboot              reboot

0 0-11 * Mon-Fri *   each hour before 12 on weekdays

Run in a Shell

  1. A shell

    /bin/sh # Bourne Shell
/bin/bash # bash
/bin/csh # c shell
/bin/tcsh # Turbo C Shell
ksh # Korn shell

  2. A script

    • generally speaking, the difference is if the script is run under current shell or not
    • read help for more
    • BTW the “.sh” at the end is not necessary
    ./script.sh
# chmod u+x ./script.sh # give user permission to execute the file if you want to run the script as the line above

# if error, check if current user has execute permission
. script.sh      # born shell?
source script.sh # c shell?
sh script.sh
bash script.sh

  3. Secure Shell

    • -C compress
      • “Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks”
    • -f SSH go background before execution
    • -n redirects stdin from /dev/null (disable stdin), ssh will go background, not work if password needed
    • -N do not execute a remote command, useful for forwarding
    • -T disable pseudo-terminal allocation, useful for forwarding
    • port forwarding 
    ssh
ssh user@0.0.0.0 -p 22 # specify user, ip address, and port

# port forwarding
## local
## localhost:8080 will be forwarded to www.example.com:80
ssh -L 8080:www.example.com:80 -L 12345:example.com:80 user@<host-used-as-jump-server>
## bind the 8080 to be accessible only from 127.0.0.1 (local)
ssh -L 127.0.0.1:8080:www.example.com:80 -L 12345:example.com:80 user@<host-used-as-jump-server>
## remote
## remote-SSH-server:5900 will be forwarded to localhost:5900
ssh -R 5900:localhost:5900 user@<remote-SSH-server>
## dynamic
## creates a SOCKS tunnel on 1080
ssh -D 1080 user@<remote-SSH-server>

Build and Install

  1. make ritual

    ./configure
make && make check
sudo make install

# or
./configure && make -j2
sudo make install

./configure --prefix=$HOME/abcdefg
make
make check
make install

  2. Libraries

    sudo ldconfig

# as root
echo /usr/local/lib > /etc/ld.so.conf.d/usr_local_lib.conf
ldconfig

  3. Strip debug codes

    strip xxx

Debugging

  1. Debugging Shell Scripts

    set -o xtrace
sh -x ./script.sh
# write "#!/bin/sh -xv" at the beginning of the script

  2. Memory leaks

    gdb
jdb
valgrind

  3. Measure Time

    time [command]
/usr/bin/time -v [command]

  4. Debuggers

    gdb
jdb

Uncanny Commands

  1. Sleep

    • Sleep for [number] seconds
    sleep [number]

# print loading... with more and more dots
echo -n "loading"
while [ '<some-status-checking-cmd> &> /dev/null; echo $?' -ne 0 ]
do
  echo -n "."
  sleep 2 # sleep for 2 seconds
done

  2. Output repeatedly until killed

    • May break the shell or ssh connection due to network lag (cannot kill)
    yes

  3. Wait

    • wait for a job to complete
    sleep 8&
# get pid of specific process
wait `ps -C sleep -o pid=`
wait `pidof sleep`

Message

  1. Turn on/off

    • It’s just turning of/off the write permission of one’s pts file
    mesg y
mesg n

  2. Send messages

    write [pts]
wall # write all

Hardware Information

  1. Get/Set SATA/IDE device parameters

    hdparm

  2. PCI devices

    • -v
    lspci

  3. USB devices

    lsusb -v
lsinput

  4. Models loaded in the Linux Kernel

    lsmod

  5. CPU

    • troubleshoot
    cat /proc/cpuinfo

  6. Mem

    • troubleshoot
    free
cat /proc/meminfo

  7. HDD

    df -h
sudo fdisk -l
hdpam -i /dev/sda[ID]

Disk and Filesystem

  1. Mount filesystems

    mount

  2. Unmount filesystems

    umount

  3. Disk management

    fdisk

  4. Remount root partition

    mount -o remount,rw /

Run levels

  1. runlevel
  2. init

Power

  1. Shutdown and reboot

    • -h hold/poweroff the machine
    • -r Reboot
    • time can be
      • now
      • +10
      • 23:10
    shutdown [-h/-r] [time]
shutdown -h
shutdown -r

poweroff # shutdown right now
reboot # reboot right now
    • Fun fact: -h is to display help usually, but if you want help from shutdown, it’s shutdown right now!

Printers

Command-Line Printing and Options

lpr -P [printer] file1 file2 file3 ...
lpr -P [printer] < file1

lpq -P [printer]

lprm -P [printer] [Jobs-id\username]

Gpg Signature

gpg

gpg --import [keyfile]
gpg2 --keyserver [URL_to_key_server] --search-keys [sender]

gpg --verify [sigfile] [file]

gpg --gen-key # generate new key
gpg --full-generate-key # generate new key pair ("full featured")
gpg --gen-revoke [KEYID] # generate revoke
gpg --send-keys [KEYID] # publish your keys to internet

gpg --fingerprint

gpg --list-keys
gpg --list-secret-keys --keyid-format LONG # list private keys

gpg --export
gpg --export -a "some user name" # -a for armored ascii
gpg --export -a "some user name" > public.key # -a for armored ascii
gpg --export-secret-key -a "some user Name" > private.key # export private key
gpg --import private-or-public.key # import key

gpg --delete-key "some user Name"
gpg --delete-secret-key "some user Name"

gpg -e -u "some user name, use this sender's key to encrypt" -r "other user name, use this receiver's public key to decypt" <some-file>
gpg -d <some-file.gpg> > <output-file-name>  # redirect to a file
gpg -o <output-file-name> -d <some-file.gpg> # -o must proceed -d

Use gpg with git

git config --global commit.gpgsign true     # turn on gpg sign
git config --global user.signingkey [KEYID] # assign gpg key

# windows gpg
git config --global gpg.program "/c/GnuPG/bin/gpg.exe"
git config --global gpg.program "C:\GnuPG\bin\gpg.exe"

git commit --amend --no-edit -n -S          # sign the last commit, maybe need to force push later
git rebase --exec 'git commit --amend --no-edit -n -S' -i [tag,hash] # another way to go?

Also config file available

~/.gnupg/gpg.conf
~/.gnupg/gpg-agent.conf

Distribution Specific

Debian

Ubuntu is based on Debian

  • What is the difference between apt and apt-get?

  • Install build essentials

    sudo apt install build-essential
sudo aptitude install build-essential

  • Unattended Upgrades

    • “keep the computer current with the latest security (and other) updates automatically”
    • at least for the security updates
    • not recommend auto-update all packages
    • UnattendedUpgrades - Debian Wiki
    sudo apt-get install unattended-upgrades apt-listchanges
sudo editor /etc/apt/apt.conf.d/50unattended-upgrades

  • upgrade

    • upgrade will potentially hold back versions and cause potential problems
    sudo apt-get dist-upgrade
# OR
sudo apt full-upgrade

  • Reconfig timezone

    sudo dpkg-reconfigure tzdata

  • List installed packages and search

    dpkg-query -l

  • Add apt source

    sudo editor /etc/apt/sources.list.d/newlist.list

  • VirtualBox - Debian Wiki

Miscellaneous

http://www.brendangregg.com/Perf/linux_observability_tools.pngLinux Observability Tools

  • Linux Performance

    • troubleshoot
    • has the prominent chart above

  • Trick to run redirect using sudo

    sudo sh -c 'echo xxx >/xxx/xxx'

  • Convert all tab to spaces

    find . -name '*.java' ! -type d -exec bash -c 'expand -t 4 "$0" > /tmp/e && mv /tmp/e "$0"' {} \;

  • Download all files from a website

    wget -A pdf,jpg -m -p -E -k -K -np <some-url>
wget --accept pdf,jpg --mirror --page-requisites --adjust-extension --convert-links --backup-converted --no-parent <some-url>

  • Calculator

    echo "scale=6;(2/3)+(7/8)" | bc

  • sass auto minify

    sass --watch a.scss:a.css --style compressed

  • Sort based on line length

    cat testfile | awk '{ print length, $0 }' | sort -n -s | cut -d" " -f2-
cat testfile | awk '{ print length, $0 }' | sort -n | cut -d" " -f2-