Unix & Linux Commands Cookbook

HUM307 Command-line Tips and Tricks from Brian Kernighan (archived) Problem solving with unix

Download a whole website bash wget -w 2 -r -np -k -p www.example.com

Extract text of a webpage bash curl www.google.com | lynx -dump -stdin -dump (extract the text), -stdin (read from the pipe)

Search directory tree for a word in file name (tree depth by 2, case insensitive) bash tree -L 2 | grep 'lisp' -i -L: (limit the depth), -i: (ignore case)

Find files modified within last 3 days bash find . -mtime -3 -mtime -3 (modified within last 3 days) - You can use +3 instead for finding files modified before last 3 days

Add same line to all files recursively in a dir bash find . -type f -name "*.md" -exec sh -c 'echo "This line will be added to all" >> "\(0"' {} \;

Move all files in subdirectories to the current path bash find . -mindepth 2 -type f -exec mv {} . \; - -mindepth 2 ensures we only get files in subdirs (not those already in current dir)

Remove all empty directories bash find . -mindepth 1 -type d -empty -delete - -mindepth 1 ensures we don't try deleting the current directory

Find large files over 100MB and sort them by size bash find . -type f -size +100M -exec ls -lh {} \; | sort -rh -k5 -size +100M (files larger than 100MB), sort -rh (reverse human-readable sort)

Find and replace text in multiple files bash find . -type f -name "*.txt" -exec sed -i 's/oldtext/newtext/g' {} +

Find duplicate files based on content (not name) bash find . -type f -exec md5sum {} \; | sort | uniq -w32 -dD

Create a simple HTTP server in current directory bash python3 -m http.server 8080

Find which process is using a port (e.g. port 8000) lsof -i :8000 - lsof > list open files - In Unix-like systems, everything is a file — this includes regular files, directories, block devices, sockets, pipes, etc. So lsof can show a wide range of useful system activity.

Check what a process does (e.g. process with id 1202212) ps aux | grep 1202212

Generate a tree view of directory excluding certain patterns bash tree -I 'node_modules|cache|tmp|vendor|.git' --dirsfirst -aC

Remove empty lines from a file bash sed -i '' '/^[:space:](/dead)*\)/d' file-path

Find all URLs in a directory, and clean them from http(s) prefixes and trailing slashes, and list all bash find . -type f -name "*.[txt|md]" -exec perl -lne 'print \(1 while /(https?:\/\/[^\s)\]]+)/g' {} \; | sed -e 's|^https://||;s|^http://||' -e 's/\/\)//'

Compare URL's in two files: find all URLs in a directory, and clean them from http(s) prefixes and trailing slashes, and list all bash capture_urls() { perl -lne 'print \(1 while /(https?:\/\/[^\s)\]]+)/g' "\)1" | sed -e 's|^https://||;s|^http://||' -e 's/\/\(//' } diff <(capture_urls file1.md) <(capture_urls file2.md) 1. diff <(...) <(...): Compare output of two commands 2. For each file: - Extract URLs using perl (perl -lne 'print \)1 while /(https?:\/\/[^\s)\]]+)/g') - Clean URLs by removing http(s) prefix and trailing slashes using sed 3. Shows differences between the two files: - Lines starting with < appear only in file1 - Lines starting with > appear only in file2 - No output means the files contain the same URLs

Copy-paste in pipeline bash pbpaste | <your-command>

Prettify youtube transcripts 1. Copy transcript to clipboard (have a format with bunch of timestamps and new lines) 2. Run the following bash pbpaste | sed 's/[0-9]:[0-9][0-9]//g' | tr -d '\n'

Unix & Linux Commands Cookbook

References