cli

2018.05.16

Query TMDB for movie and tv info on the commandline

Christopher Peterson Blog cli, code, Ruby

You know TMDB (The Movie DataBase)? It’s like IMDB, but without being awful.

tl;dr: I made a cli TMDB query tool in Ruby and here it is with documentation: https://github.com/decipher-media/tmdbget

As part of our podcast website migration from WordPress to Jekyll, I’m coding in a bunch of functionality that I’ve wanted for a while. The first one: a cli utility to fetch movie metadata into (semi-)structured data.

So here it is

https://github.com/decipher-media/tmdbget

Grab some API keys form TMDB and it’s easy-peasy to get some movie and tv metadata:

bundle exec tmdbget.rb --key [TMDB API KEY] 'the terminator'

1	bundle exec tmdbget.rb --key [TMDB API KEY] 'the terminator'

To get something like:

{
  "The Terminator": {
    "vote_count": 5259,
    "id": 218,
    "video": false,
    "vote_average": 7.4,
    "title": "The Terminator",
    "popularity": 21.31828,
    "poster_path": "/q8ffBuxQlYOHrvPniLgCbmKK4Lv.jpg",
    "original_language": "en",
    "original_title": "The Terminator",
    "genre_ids": [
      28,
      53,
      878
    ],
    "backdrop_path": "/6yFoLNQgFdVbA8TZMdfgVpszOla.jpg",
    "adult": false,
    "overview": "In the post-apocalyptic future, reigning tyrannical supercomputers teleport a cyborg assassin known as the \"Terminator\" back to 1984 to kill Sarah Connor, whose unborn son is destined to lead insurgents against 21st century mechanical hegemony. Meanwhile, the human-resistance movement dispatches a lone warrior to safeguard Sarah. Can he stop the virtually indestructible killing machine?",
    "release_date": "1984-10-26",
    "release_year": "1984"
  }
}

{

"The Terminator": {

"vote_count": 5259,

"id": 218,

"video": false,

"vote_average": 7.4,

"title": "The Terminator",

"popularity": 21.31828,

"poster_path": "/q8ffBuxQlYOHrvPniLgCbmKK4Lv.jpg",

"original_language": "en",

"original_title": "The Terminator",

"genre_ids": [

28,

53,

878

"backdrop_path": "/6yFoLNQgFdVbA8TZMdfgVpszOla.jpg",

"adult": false,

"overview": "In the post-apocalyptic future, reigning tyrannical supercomputers teleport a cyborg assassin known as the \"Terminator\" back to 1984 to kill Sarah Connor, whose unborn son is destined to lead insurgents against 21st century mechanical hegemony. Meanwhile, the human-resistance movement dispatches a lone warrior to safeguard Sarah. Can he stop the virtually indestructible killing machine?",

"release_date": "1984-10-26",

"release_year": "1984"

}

Check the help output for full and up-to-date functionality details, including a flag for interactive mode for when you’re less than confident that you’ll get the right answer. 🤓

2018.02.16

How we produce our podcast on Linux at Decipher SciFi

Christopher Peterson Blog Audacity, BASH, cli, FFmpeg, Instagram, Linux, podcasting

You may be a Linux podcasting person looking for some ideas after a couple of years of Linux podcast production. You may be a fan of Decipher SciFi who wants to see what it looks like on the back end. In either case, welcome!

This is not meant to be the guide to doing it right, but just a record of the hackey way that we do it for reference. It’s like looking at someone’s Vim config; you don’t just copy the thing, but take what is useful.

Our usual recording setup is the two of us in the same room with sporadic remote guest appearances via Skype. This is the stuff we use and how (links to products on Amazon may include an affiliate code 🙂 ).

Recording at the “studio” with Jolene Creighton for our Star Trek Beyond episode

Recording

Hardware

Microphones (ATR-2100)

The ATR-2100 microphone can use both USB and XLR

My partner and I both use ATR-2100 microphones. We settled on these for a few reasons:

Each has both XLR and USB interfaces, so we were able to continue to use them when graduating from computer to dedicated recorder.
They’re not overly sensitive so they work well when recording two people in the same room.
They’re affordable!

But, there is another. Since our mics and the system output from the Skype machine both go directly into the recorder, we need to have another microphone for our guests to hear us. We keep a Blue Yeti (which we do not recommend for the actual podcast recording) sitting in the middle of the table for this purpose, but whatever you have will probably do.

Recorder (Zoom H5)

We started out recording directly into the computer with Audacity which proved to ultimately be less than reliable. Maybe this could be a fine enough affordable option for one person, I suppose, but if you’re recording two people in the same room you have a lot of pain-in-the-ass fiddling with audio configurations to look forward to, iirc (to aggregate separate USB audio sources). After a while and a few disastrous software failures in this mode, we finally got a dedicated recorder. In fact, I would actually recommend a dedicated recorder no matter what platform you’re on.

Zoom H5 recorder

So, this thing. This thing is pretty great! It can record four channels – two single-channel XLR inputs on the bottom, and a combined stereo input from whatever “capsule” is plugged into the top. With the X/Y capsule (it ships with this one) on top we are able to plug in remote guests from a Skype session on the nearest computer and record all three tracks separately.

Our recording setup

Editing

Okay, finally on to some software stuff.

Convert

The H5 records in WAV, and can basically only record input from the capsule (Skype) in stereo. So I want to get these tracks into mono FLAC files because it is lossless without being huge.

Here is how I do this conversion (the current “production” version of this script can be found in my dotfiles).

#!/usr/bin/env bash                                                                                                     
                                                                                
# Requires:                                                                     
# ffmpeg                                                                        
                                                                                
# Usage                                                                         
if [ $# -eq 0 ]; then                                                           
  cat <<-EOF                                                                    
  Usage: $0 episode_dir file1 file2 file3...                                    
  Convert a bunch of WAVs to FLAC with names to match the target dir.           
  (For converting WAVs from the Zoom H5 into podcast tracks for editing by my   
  naming scheme)                                                                
                                                                                
    episode_dir | A directory named as the title of a podcast episode.          
    file1...n   | WAV files named with either a number or 'LR' as produced by   
                  the Zoom H5 recorder.                                         
EOF                                                                             
  exit 1                                                                        
fi                                                                              
                                                                                
episode_dir=$1                                                                  
files=("${@:2}")                                                                
moviename=$(basename "$episode_dir" | sed 's/^[0-9 ]*//')                       
                                                                                
mkdir -p "$episode_dir"                                                         
                                                                                
for file in "${files[@]}"; do                                                   
  num=$(echo "$file" | grep -Po '([0-9]|LR)(?=\.[wW][aA][vV])')                 
  num=$(echo "$num" | sed 's/LR/0/')                                            
  ffmpeg \                                                                      
    -i "$file" \                                                                
    -ac 1 \                                                                     
    "${episode_dir}/${moviename}.raw.${num}.flac"                               
done

#!/usr/bin/env bash

# Requires:

# ffmpeg

# Usage

if [ $# -eq 0 ]; then

cat <<-EOF

Usage: $0 episode_dir file1 file2 file3...

Convert a bunch of WAVs to FLAC with names to match the target dir.

(For converting WAVs from the Zoom H5 into podcast tracks for editing by my

naming scheme)

episode_dir | A directory named as the title of a podcast episode.

file1...n | WAV files named with either a number or 'LR' as produced by

the Zoom H5 recorder.

EOF

exit 1

episode_dir=$1

files=("${@:2}")

moviename=$(basename "$episode_dir" | sed 's/^[0-9 ]*//')

mkdir -p "$episode_dir"

for file in "${files[@]}"; do

num=$(echo "$file" | grep -Po '([0-9]|LR)(?=\.[wW][aA][vV])')

num=$(echo "$num" | sed 's/LR/0/')

ffmpeg \

-i "$file" \

-ac 1 \

"${episode_dir}/${moviename}.raw.${num}.flac"

done

Edit

Then I import the tracks into Audacity and go to town.

Here is my Audacity configuration and here are the plugins I use.

Post production

With the edit done, I can then run it through a final set of filters and cut in some intro and outro music (generously provided by Caelum Rale). Final export to 112kbps mono CBR LAME MP3 (plenty good enough for voice) with our id3 tags, add the thumbnail (Audacity doesn’t support this), and we’re done.

Final filters

We use Levelator because it works, still, and is at least free as in beer. It’s super old and out of support but it works just fine for us, so here it stays. You can get Levelator here, but the link to the Linux version is broken. The Windows version works superbly in Wine though!

Levelator is pretty easy to use

The alternative these days seems to be Auphonic, a web-based service that gives you a few free sessions per month or somesuch and is reasonably priced for more. It seems popular and I hear good things.

Thumbnail/cover art

Audacity doesn’t know how to write the cover art id3 tags, but it’s easily done on the cli anyway. eyeD3 is a good tool for this.

eyeD3 --add-image "/path/to/my/id3-tinylogo.jpg:FRONT_COVER" DecipherSciFi.mp3

1	eyeD3 --add-image "/path/to/my/id3-tinylogo.jpg:FRONT_COVER" DecipherSciFi.mp3

It’s not a big deal, but it’s nice to have.

Publishing

File Hosting

Our finalized MP3s get uploaded to Libsyn. Another good option I am happy to recommend is Blubrry. Either way, as an enterprising Linux nerd, you’d only be using it for file hosting because you’ll control the RSS feed itself from your own site? Right?

All kidding aside, people in the space continue to fight over whether you should control the feed from your own domain or relinquish control to the hosting service itself. You do you, but I prefer the former. And there’s a great WordPress plugin for that too mentioned below!

Website hosting

Our post for the episode then goes up on Decipher Scifi. The one good option here is WordPress(.org), really. How and where you do this is up to you but we run a multisite WordPress install on Linux on AWS and manage it ourselves. I do have a couple of tips re link shortening and Let’s Encrypt though if you’re into that sort of thing.

And then within WordPress, we use the only (and really good!) plugin for serving up podcast RSS feeds, Blubrry PowerPress. It’s never done us wrong.

Backups

Yeah don’t forget these! Both the audio and the website.

Audio backups

When the episode is up and out, we back up the raw tracks and the separate final edits of the tracks in FLAC, and the final cut in MP3. They go to Dropbox, Google Drive, S3, whatever, and a portable hard drive or two. The idea is to have both a local and remote copy if you value your backups.

The final edit files for one of our episodes

Website backups

I wrote a small hacky BASH script that does my WordPress backups. It tars and gzips both the entire WordPress directory and the database dump and then moves it around with Rsync. One day I’ll do the work of finding/creating a better, more robust solution. Let me know if you can recommend one?

Social

We have social channels! Twitter | Facebook | Instagram

Creating posts

Simple text posts are one thing, but I like to use images or video when possible. I use the following tools to help me with this.

GIMP. Here is my GIMP config, if you’re interested. As of writing, It is set to behave mostly sorta roughly like Photoshop, but I still haven’t tuned this up as much as I’d like. Caveat emptor.

Instagif is a script I wrote to convert GIF (from local or the web) to MP4 for Instagram. Facebook and Twitter both support GIF (though iirc they ultimately convert to MP4 anyway), but you need to handle Instagram differently and make the GIF into an MP4 that meets with Instagram’s length and color encoding requirements. Thus, the script. 🙂

Maybe I should just skip straight from GIF to MP4 for all the platforms? I’ll need to test to see if the viewing experience is diminished by this switch on Facebook and Twitter. Will report back.

Posting posts

Hootsuite!

The free tier of this wonderful service allows for scheduling up to 30 posts, which is more than I ever do at one time. And now it can even take advantage of Instagram’s direct-posting feature for business accounts, if you have one.

Summary

Hardware

Zoom H5 Recorder with a fast SD card
ATR-2100
Neewer NW-35 table-mounting mic arm. Super cheap and has been serving us with various heavy microphones for a good long while. It even holds up the super-heavy Blue Yeti which we used for a minute.

Software

Services

Decipher SciFi Bill & Ted selfie with Brandon Rollins

podcast on, dudes!

So there you have it. Amazon product links include our affiliate code, if you don’t mind helping us out a bit. Happy Linux podcasting! 😀

2018.02.13

Convert GIF to MP4 for Instagram with FFmpeg

Christopher Peterson Blog BASH, cli, code, FFmpeg, Instagram

I made a script to account for Instagram’s idiosynchracies when converting a GIF to MP4 using FFmpeg. Here it is:

https://github.com/cspeterson/dotfiles/blob/master/.bin/instagif

And now the details…

Doing a conversion of GIF to MP4 with ffmpeg seems like it should be simple enough:

ffmpeg -i something.gif out.mp4

1	ffmpeg -i something.gif out.mp4

But it isn’t! This can be insufficient in a couple of ways..

Problem 1: Video too short

Solution: Use a filter to loop the input enough times to meet the 3s minimum time requirement

-filter_complex "loop=<NUMBER_OF_LOOPS>"

1	-filter_complex "loop=<NUMBER_OF_LOOPS>"

Problem 2: Wrong color encoding

Given a color encoding that it doesn’t understand, Instagram just kinda poops out

Solution: By default, my ffmpeg used yuv444p, which Instagram wasn’t happy with. I haven’t done an exhaustive survey of the color encoding that Instagram will accept, but here is one: yuv420p.

-pix_fmt yuv420p

1	-pix_fmt yuv420p

In addition, the conversion requires the file’s height to be divisible by 2, so we need yet another filter:

-filter_complex "scale=trunc(iw/2)*2:trunc(ih/2)*2"

1	-filter_complex "scale=trunc(iw/2)2:trunc(ih/2)2"

Automation

Now since so many GIFs that I wish to post to Instagram are actually shorter than 3s, I automated everything above and here is the first cut of the script. To see if I made any changes since actually using it “in production” check the version I’m currently using in my dotfiles.

#!/usr/bin/env bash

# Required:
# bc
# ffmpeg
# wget

set -e

# Usage
if [ $# -eq 0 ]; then
  cat <<-EOF
  Usage: $0 infile outfile
  Convert a gif to mp4 with ffmpeg, looping it enough times to ensure it meets
  Instagram's minimum video length limit.

    infile  | A valid gif file to convert. If given a URI, this script will
              try to download it for you and then convert it.
    outfile | A target filename for the result. If the output filename is not
              specified, it will be placed alongside the input file with the
              extension '.insta.mp4' added.
EOF
  exit 1
fi

# MAIN
INFILE=$1
OUTFILE=$2
MINTIME=3

# Download the input file if given a URI
urire='(https?|ftp|file)://[-A-Za-z0-9\+&@#/%?=~_|!:,.;]*[-A-Za-z0-9\+&@#/%=~_|]'
if [[ $INFILE =~ $urire ]]; then
  filename=$(basename $INFILE)
  wget -O "${filename}" "${INFILE}"
  INFILE="${filename}"
fi

# Determine output filename
if [ -z $OUTFILE ]; then
  OUTFILE=$(echo "$INFILE" | sed 's/\.gif$//')
  OUTFILE="${OUTFILE}.insta.mp4"
fi

# Check if input is gif
if [[ $(file $INFILE) != *GIF* ]]; then
  echo "Input file '$INFILE' is not a gif. Quitting."
  exit 1
fi

# Get info on the gif
# (A simple -i or ffprobe would be sufficient to get the information we need for
# a video file, but not a gif. Basically as far as I can tell you need to
# process the file in a manner similar to this to get the frame count, and it
# then also provides us with the fps which we need anyway)
ffoutput=$(ffmpeg -i "$INFILE" -f null /dev/null 2>&1)
framecount=$(echo "$ffoutput" | grep -Po 'frame=\s+[0-9]+\s+' | egrep -o '[0-9]+')
vstreaminfo=$(echo "$ffoutput" | grep -P 'Video:\s+gif')
fps=$(echo "$vstreaminfo" | grep -Po '[0-9]+(?=\s+fps)')
# dims=$(echo "$vstreaminfo" |  grep -Po '[0-9]+x[0-9]+')
# w=$(echo "$dims" | cut -d 'x' -f 1)
# h=$(echo "$dims" | cut -d 'x' -f 2)
len=$(echo "$framecount / $fps" | bc -l | sed 's/0\{1,\}$//')
if (( $(echo "$len >= 60" | bc -l ) == 1 )); then
  echo "Clip is too long for Instagram! The limit is 60s. Quitting."
  exit 1
fi

# How many loops do we need to meet instagram's min time? Round up with awk.
loops=$(echo "$MINTIME / $len" | bc -l | awk '{print ($0-int($0)>0)?int($0)+1:int($0)}')

# Convert
ffmpeg -i "$INFILE" \
       -filter_complex "loop=${loops}:32767:0,scale=trunc(iw/2)*2:trunc(ih/2)*2" \
       -f mp4 \
       -y \
       -preset slow \
       -pix_fmt yuv420p \
       "$OUTFILE" >/dev/null 2>&1

#!/usr/bin/env bash

# Required:

# bc

# ffmpeg

# wget

set -e

# Usage

if [ $# -eq 0 ]; then

cat <<-EOF

Usage: $0 infile outfile

Convert a gif to mp4 with ffmpeg, looping it enough times to ensure it meets

Instagram's minimum video length limit.

infile | A valid gif file to convert. If given a URI, this script will

try to download it for you and then convert it.

outfile | A target filename for the result. If the output filename is not

specified, it will be placed alongside the input file with the

extension '.insta.mp4' added.

EOF

exit 1

# MAIN

INFILE=$1

OUTFILE=$2

MINTIME=3

# Download the input file if given a URI

urire='(https?|ftp|file)://[-A-Za-z0-9\+&@#/%?=~_|!:,.;]*[-A-Za-z0-9\+&@#/%=~_|]'

if [[ $INFILE =~ $urire ]]; then

filename=$(basename $INFILE)

wget -O "${filename}" "${INFILE}"

INFILE="${filename}"

# Determine output filename

if [ -z $OUTFILE ]; then

OUTFILE=$(echo "$INFILE" | sed 's/\.gif$//')

OUTFILE="${OUTFILE}.insta.mp4"

# Check if input is gif

if [[ $(file $INFILE) != *GIF* ]]; then

echo "Input file '$INFILE' is not a gif. Quitting."

exit 1

# Get info on the gif

# (A simple -i or ffprobe would be sufficient to get the information we need for

# a video file, but not a gif. Basically as far as I can tell you need to

# process the file in a manner similar to this to get the frame count, and it

# then also provides us with the fps which we need anyway)

ffoutput=$(ffmpeg -i "$INFILE" -f null /dev/null 2>&1)

framecount=$(echo "$ffoutput" | grep -Po 'frame=\s+[0-9]+\s+' | egrep -o '[0-9]+')

vstreaminfo=$(echo "$ffoutput" | grep -P 'Video:\s+gif')

fps=$(echo "$vstreaminfo" | grep -Po '[0-9]+(?=\s+fps)')

# dims=$(echo "$vstreaminfo" | grep -Po '[0-9]+x[0-9]+')

# w=$(echo "$dims" | cut -d 'x' -f 1)

# h=$(echo "$dims" | cut -d 'x' -f 2)

len=$(echo "$framecount / $fps" | bc -l | sed 's/0\{1,\}$//')

if (( $(echo "$len >= 60" | bc -l ) == 1 )); then

echo "Clip is too long for Instagram! The limit is 60s. Quitting."

exit 1

# How many loops do we need to meet instagram's min time? Round up with awk.

loops=$(echo "$MINTIME / $len" | bc -l | awk '{print ($0-int($0)>0)?int($0)+1:int($0)}')

# Convert

ffmpeg -i "$INFILE" \

-filter_complex "loop=${loops}:32767:0,scale=trunc(iw/2)*2:trunc(ih/2)*2" \

-f mp4 \

-y \

-preset slow \

-pix_fmt yuv420p \

"$OUTFILE" >/dev/null 2>&1

Usage

To convert yup.gif to yup.mp4

instagif yup.gif yup.mp4

1	instagif yup.gif yup.mp4

To convert yup.gif into yup.insta.mp4

instagif yup.gif

1	instagif yup.gif

To create an insta.mp4 directly from a remote GIF

instagif https://media.giphy.com/media/lgcUUCXgC8mEo/giphy.gif insta.mp4

1	instagif https://media.giphy.com/media/lgcUUCXgC8mEo/giphy.gif insta.mp4

Happy giffing! 😀

2017.11.25

A faster LastPass lookup for your Linux desktop

Christopher Peterson Blog BASH, cli, dmenu, Firefox, LastPass, Linux

At work we noticed how the LastPass plugin on the new Firefox Quantum (due to limitations of the new plugin architecture?) no longer has copy a “Copy Password” button. LastPass support suggested that this might not be changing anytime soon. We rely on this a lot for different terminal-based work, so it was a sad revelation.

This bothered me enough that I made my own LastPass popup thing for my Linux desktop and as a bonus the workflow is also much faster than it was in the browser.

https://github.com/cspeterson/dotfiles/blob/master/.bin/lastpass-dmenu

Here is the code in case I move the file or something and forget to update this post:

IFS=$'\n'
# List all entries in LastPass vault into dmenu formatted as follows
# Folder/subfolder/Name of Site [username at site] [id: id for lookup]
entries=($(lpass ls --long \
  | cut -d ' ' -f 3- \
  | sed 's/\[username: /[/' \
  | sed 's/\(.*\)\(\[.*\]\) \(\[.*\]\)/\1 \3 \2/')
  )

# Get id from dmenu user selection
selid=$(printf '%s\n' "${entries[@]}" \
  | dmenu -i -p 'LastPass: ' -l 7 \
  | sed 's/^.*\[id: \([0-9]\{1,\}\)\].*$/\1/')

# Password to clipboard
lpass show --clip --password ${selid}

IFS=$'\n'

# List all entries in LastPass vault into dmenu formatted as follows

# Folder/subfolder/Name of Site [username at site] [id: id for lookup]

entries=($(lpass ls --long \

| cut -d ' ' -f 3- \

| sed 's/\[username: /[/' \

| sed 's/$.*$$\[.*\]$ $\[.*\]$/\1 \3 \2/')

)

# Get id from dmenu user selection

selid=$(printf '%s\n' "${entries[@]}" \

| dmenu -i -p 'LastPass: ' -l 7 \

| sed 's/^.*\[id: $[0-9]\{1,\}$\].*$/\1/')

# Password to clipboard

lpass show --clip --password ${selid}

Documentation for the script is in the file, but here is exactly what install could look like on Ubuntu, for example

sudo apt-get update
# Lastpass cli is the lastpass cli client from LastPass themselves
sudo apt-get install dmenu lastpass-cli
mkdir $HOME/.bin
wget -O $HOME/.bin/lastpass-cli https://raw.githubusercontent.com/cspeterson/dotfiles/master/.bin/lastpass-dmenu
chmod +x $HOME/.bin/lastpass-cli
# Login to lastpass-cli one time and it will remember your email for the
# future
lpass login myuser@ias.edu

sudo apt-get update

# Lastpass cli is the lastpass cli client from LastPass themselves

sudo apt-get install dmenu lastpass-cli

mkdir $HOME/.bin

wget -O $HOME/.bin/lastpass-cli https://raw.githubusercontent.com/cspeterson/dotfiles/master/.bin/lastpass-dmenu

chmod +x $HOME/.bin/lastpass-cli

# Login to lastpass-cli one time and it will remember your email for the

# future

lpass login myuser@ias.edu

And then run it there(no) or bind it to a hotkey(yes) in your window manager/whatever (I think this is under Keyboard settings in e.g. Gnome?).

When the menu pops up, just start typing the entry you want. It searches through both entry Name and Username. Whichever entry you select in the dmenu popup, the corresponding password will drop into your clipboard. Easy peasy.

Security

By default as of this writing, lpass seems to just leave your password in the X primary clipboard forever (or until overwritten). It recognizes an environment variable however, LPASS_CLIPBOARD_COMMAND, where you can specify your clipboard command and arguments. This allows for a setting like the following

export LPASS_CLIPBOARD_COMMAND="xclip -selection clipboard -in -l 1"

1	export LPASS_CLIPBOARD_COMMAND="xclip -selection clipboard -in -l 1"

Which will allow one X selection request (i.e. a paste action) before this value is cleared from the clipboard. Hopefully the default will change in the future to be more secure? But there go in the meantime.

<edit> After further consideration it seems the environment variable trickery above will remain the only solution, so get used to it 😉