fragmented knowledge

bruchstückhaftes Wissen

Schnipsel

Compress movies in parallel to vp9/webm

parallel --eta -j4 'ffmpeg -y  -i {}  -c:v libvpx-vp9 -crf 38 -threads 4 -b:v 0 -c:a libopus -b:a 64000  {.}.webm' ::: $(ls -1 *mp4)

Save installed packages and restore them on another Debian system

source

dpkg --get-selections > packages.txt

destination

sudo dpkg --set-selections < packages.txt
sudo apt-get dselect-upgrade --with-deps

Log memory consumption (USS, PSS, RSS)

for i in {1..100000} ; do date >> mem.log ;  smem -a -p -s uss -r | head -n 8 >> mem.log ; sleep 0.2 ; done

create MikroTik certificate bundle with intermediate CA

cat "$cert_file" ca-intermediate.pem "$priv_file"  > $router-name.pem

integer overflow in awk

On Debian mawk is the default awk implementation, but mawk only works with 32-bit long data types. If you need more, you have to use gawk with the --bignum option. The example sums up the first column of a file.

gawk '{ sum += $1 } END { printf "%d\n", sum }'

Warning: In my example gawk is 40% slower and with --bignum activated it is more than 100% slower.

getting list of file sizes for an extreme amount of files

This is much faster than using find’s -exec, because -exec forks a du for every input line.

find . -type f | xargs -d '\n' -n 65535 du -b --apparent-size > file_sizes

listing active users (local)

users=$(egrep '^.*:\$.*$' /etc/shadow | cut -d ":" -f 1) ; for u in $users ; do lastlog -u $u | tail -n +2 | grep -v root ; done

detect log4shell with trivy on active docker images

for image in `docker ps --format "{{.Image}}"` ; do echo -n "$(hostname): $image "; trivy image  --skip-update -s CRITICAL $image | grep "CVE-2021-44228"; echo ; done |  grep "CVE-2021-44228" | cut -d "|" -f 1

pdf cut

Cut pages from 363 to 378 from proceedings.pdf to paper.pdf:

qpdf --collate --empty --pages proceeding.pdf 363-378 -- paper.pdf

kvm and qemu

Create qcow2 image which depends on a base image

qemu-img create -F qcow2 -b BASE.qcow2 -f qcow2 NEW.qcow2

Commit changes from the NEW image to the BASE image

qemu-img commit -f qcow2 -b BASE.qcow2 NEW.qcow2

bash

sorting a tsv file by its second column (biggest value first)

sort  -t$'\t' -k2 -n -r

read from a newline-separated file and rearrange lines into three columns

paste - - -

show matching lines from two files

grep -F -f file1 file2

Take the lines from file1 (-f) and use them as a raw string pattern (-F) and grep file2.

midi -> mp3

timidity file.mid -Ow -o - | ffmpeg -i - -acodec libmp3lame -ab 128k output.mp3

docker

autostart container:

docker update --restart=always 0576df221c0b

e-mail from CLI via remote SMTP server

using swaks

echo "Das ist eine Nachricht" | swaks --to martin@nachtsieb.de \
--from FROM_EMAIL --server FQDN:PORT --auth PLAIN \
--auth-user "USERNAME" --auth-password "SECRET" \
-tls --header "Subject: foo" --body -

PostgreSQL

show all stored procedures/user defined functions with the pattern anon42:

\df *anon42*

date conversion: epoch time in UTC -> date string

date -ud @1606482000
Fr 27. Nov 13:00:00 UTC 2020

professional downloading with youtube-dl

Downloading the (most AVC/H.264) video track with the given height in pixels and merge (+) it with the best audio track (aac) provided in an a4m container. The default behaviour is: ‘bestvideo+bestaudio’.

youtube-dl -f 'bestvideo[height=1080][ext=mp4]+bestaudio[ext=a4m]' URL

Maven

build simple maven project

mvn archetype:generate -DgroupId=XXX -DartifactId=YYY -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4

invoke single test method within test class

mvn -Dtest=TestClass#testMethod test

validate XML against DTD

xmllint -noout --dtdvalid recipes.dtd recipes.xml

disable mouse acceleration

get mouse description

xinput list

list mouse properties

xinput list-props 'USB Optical Mouse'

disable acceleration

xinput --set-prop 'USB Optical Mouse' 'libinput Accel Profile Enabled' 0, 1

svn: create ignore rules

Ignore rules effecting directories and only suport globbing (wildcard *)

svn propedit svn:ignore ./path

rules:

*.out
*.aux
*.toc
*.swp
*.tmp
*.log
*.out
*.exe
*.gz
*.tgz
*.backup
_minted*
*.o
*.bbl
*.blg
*.class
*.pdf

shrink images (prevent data trash)

for i in `ls -1` ; do convert $i  -quality 90 -resize 3200  $i ; done

in parallel with GNU parallel:

parallel --eta 'convert {} -quality 90 -resize 3200' {} ::: *jpg

epub optimization for mobile e-ink reader

for i in `ls -1` ; do convert $i -colorspace Gray  -quality 80 -resize 50%  $i ; done

remote Wireshark per SSH

ssh [TARGET] sudo tcpdump -i [INTERFACE] -U -w -port 22 | wireshark -i - -k

sort and rename files according to atime (bash)

IFS=$'\n' ; i=0 ; for file in `ls -1 -u --sort=time -r` ; do let i++; \
newfile=$(printf "%03d-%s" $i $file); mv $file $newfile ; done

SSH Portforwarding

local

I have no direct connection to binfalse.de:443, I have to tunnel through example.com.

CLIENT <--> SSH-GATEWAY <--> SERVER
^^^^^^		example.com		 binfalse.de

ssh -L 9000:binfalse.de:443 user@example.com
ssh -L LOCALPORT:DESTADDR:DESTPORT SSH-GATEWAY

remote

I tell the SSH gateway to open up a port and forward this one to my machine.

CLIENT <--> SSH-GATEWAY <--> BigBadInternet
tcp/3000  example.com:9000

ssh -R 9000:localhost:3000 user@example.com

no tty

ssh -nNT -L 9000:binfalse.de:443 user@example.com

concatenate pdf files

pdftk *.pdf cat output onepdf.pdf

extract partition from image

sfdisk -l -uS image-file
dd if=image-file of=partition-file skip=NUM count=NUM

Linux Ramdisk

mount -o size=1G -t tmpfs none /mnt/tmpfs

awk oneliners

http://www.pement.org/awk/awk1line.txt

Linux iptables: statisches NAT

iptables -t nat -A PREROUTING -d 17.17.17.50 -i eth0 -j DNAT --to-destination 10.200.0.1
iptables -t nat -A POSTROUTING -s 10.200.0.1 -o eth1 -j SNAT --to-source 17.17.17.50

Linux iptables: Portforwarding

iptables -t nat -A PREROUTING -p tcp -i eth2 -d 10.11.20.25 --dport 2222  -j DNAT --to 10.0.0.4:22

IPv6 privacy extensions unter Linux aktivieren (RFC3041)

sysctl net.ipv6.conf.wlan0.use_tempaddr=2

ASCII-armored PGP with Sylpheed

Der MUA Sylpheed ist nicht in der Lage ASCII-armored GPG/PGP-Nachrichten zu entschlüsseln. Man kann jedoch mit einer selbstdefinierten “Aktion” Abhilfe schaffen:

urxvt -e /bin/bash -c "/usr/bin/gpg -d %p | less"

convert a hex string to ASCII, using Perl

perl -e 'chomp($HEX=); while($i < length($HEX)) {$PART=substr($HEX,$i,2);print chr(hex($PART)); $i=$i+2;}'

bash IFS-variable, only split on Newline

export IFS=$'\n'