Saving YouTube Video

Trying to collect the old episodes of NYPD Blue, and there were some I couldn’t find on usenet. Turns out that the DVDs past season 4 were never produced. But there are old episodes, mostly captured from VHS that are out on Youtube.
After a little research, there is a cute way to capture the video files from youtube.
Click the episode on youtube to start it, and then capture the url in the browser location bar.
Now open VLC, select to play a network stream, and give it that url. It will start to play.
At this point, one can just tell VLC to convert and save, but that takes a long time. An easier and faster technique, as long as the youtube webm format is ok:
In vlc, with the stream playing, open tools->codec information and copy the link that is in the “Location” text box at the bottom.
Paste that link into a browser and go, and now the video plays directly from the Youtube servers.
Now you can just right click on the browser window where it is playing, and save video as.

Automatic TV Downloads

Having spent a lot of time downloading tv shows from primewire, and keeping a spreadsheet up to date about what has been gotten and what has not, I decided to look into using sickbeard to automate.

The parts of this are:

Usenet – One apparently has to have access to usenet to do this well. Usenet (sometimes also called net news) consists of thousands of “newsgroups”, and posts within the newsgroups are “articles” (originally this was all textual bulletin board type stuff). A single piece of a binary is posted in a article, so a number of articles must be reassembled to obtain the entire file. Usenet was at one time, before the invention of the world-wide web, one of the two or three most important things in the internet (with FTP and mail). One used to find a site that offered a usenet feed, and tap into it. These days apparently only some parts of usenet are available for free. The so-called “binary” newsgroups, which have all the media stuff, are too big, too volatile, and take too much bandwidth for the ISPs to provide them for free. One has to pay somebody (apparently) who does the daily updates. So I signed up for an account with Giganews.

NZB files – These are files that describe where one can find the binaries for the tv shows, or movies or what have you. They are somewhat like torrent files. The NZB file lists which articles in which news groups have the pieces for a particular file. For a given show if you can download an NZB file and give it to a download program (like sabnzbd mentioned in a moment), that program will download all the pieces, in parallel, put them together and recreate the desired file.

Sickbeard – a python script that automates the process of keeping track of what tv episodes are available for designated shows, and which ones you have. It takes advantage of indices on the net to figure out where stuff can be downloaded from usenet. The index tells which articles in which newsgroups have which piece. Sickbeard doesn’t actually do downloads. It retrieves NZB files from the index sites, and gives them to Sabnzbd.

Sabnzbd – This is a download program for downloading NZB files. It is a python script. If you give it an NZB file, it will download the parts of the desired file and reassemble them, taking care of compression, encryption, whatever, with options for renaming, removing temporary files, and lots of other stuff. This is a VERY useful thing in it’s own right. There is a wealth of stuff out there on usenet in the alt.binary newsgroups – applications, movies, tv, etc. It is quite easy to search for something, download an NZB file and stick it into the folder where Sabnzbd is listening (mentioned more later), and it will take care of everything.

Couchpotato – Where Sickbeard manages tv, Couchpotato manages movies. It keeps track of what movies are released on DVD, and when movies become available it will download them, either from usenet or via torrents from movie sites.

I have created a directory for doing all the parts of the automated tvdownloading on oregano. I put it on the /backup volume because there is a lot of space there. The torrent downloads are already there (in /torrentdownloads), so now there is a directory /tvdownloads.

I got sickbeard from git://github.com/midgetspy/Sick-Beard.git. I run the python scripts directly from the git repository. It’s configuration is in /etc/sickbear. I created /etc/systemd/system/sickbeard.service (from an example file called init.systemd). systemctl enable sickbeard, and systemctl start sickbeard.

Manage sickbeard through a web interface to http://localhost:8081. Sickbeard is started with its datadir pointing at /etc/sickbeard. In there sickbeard keeps its database, with information about each show it is managing, and the location where the finished shows should be placed (which I have pointing to my /tvshows directory). Sickbeard is configured so that a new episode becomes available, it searches usenet index sites and gets an NZB file for that episode and passes it to sabnzbd for download. When sabnzbd has completed the download, it invokes a script in sickbeard’s /etc/sickbeard/autoProcessTV folder called sabToSickBeard.py. That script takes care of taking the file downloaded by sabnzbd and moving it to /tvshows.

I had to sign up with a commercial usenet provider. I chose one called giganews. This will cost $14.95/mo.

I cloned the product SABnbzd from github (https://github.com/sabnzbd/sabnzbd.git). This is also python, and I run it also directly from the repo. It normally will create a config in /home but I set up its config in /etc/sabnzbd. I also set it up in systemd. The launch tells it where its config is. When sabnbzd runs it responds on http://localhost:8080 so it can be configured and managed with a browser. It has directories sabnzbd/incomplete and sabnzbd/complete in the /backup/tvdownloads folder to do its work in.

Sabnzbd receives its instructions about what to download in one of two ways: (a) It watches for nzb files in /backup/tvdownloads/nzbtodownload. Anything that appears there it tries to download, and it leaves the result in it’s “complete” folder; or (b) Sabnzbd generates keys (API key and NZB key) which allow other programs to communicate with it through its url (on localhost), so a requester (like sickbeard) can communicate the nzb information by connecting to Sabnzbd on the url. I have tried both, and sickbeard is currently using the url upi. Sabnzbd can also be configured to run a script when it has finished the download, and as already mentioned, it invokes a sickbeard script.

Sickbeard searches several online indexes to discover NZB files. It comes configured to use “Wombie’s Index” and the “Sickbeard Index”. It also will use some others. One of them called “Usenet-Crawler” I registered with.

I had a couple sickbeard had trouble with, don’t know why yet. But worth writing down that there are some other sites out there where you can search manually, and download nzb files by hand. For example, I downloaded NZB files manually for Hill Street Blues.

Sickbeard is now successfully monitoring the tvshows I watch. When a new episode is available it has successfully notified Sabnzbd who has downloaded the episode into its completed folder. It then invokes sickbeard’s script, and the show gets moved and renamed into the correct location in /tvshows.

I have also done “catchup” on shows. I had not downloaded any episodes of Doctor Who for the last (current) season. This morning I added Doctor Who as a show to monitor, told it to skip seasons 1-8 (which I have), but that I wanted all the episode in season 9. It immediately queued it all up and started grabbing the episodes and downloading them.

I also told Sickbeard I wanted NYPD Blue, and it managed to get almost all of it, although I had to intervene a little manually. This is a very old show, and not all the parts were available. But sickbeard did an amazing job of finding and gathering everything that was available, with a little bit of help.

I cloned couchpotato, also python, from git (https://github.com/CouchPotato/CouchPotatoServer.git) and run it from the repo. Its config is in /home/dee/.couchpotato. It listens on localhost:5050. Like sickbeard it searches sites for movies and it can use either sabnzbd (for usenet) or transmission (for torrents) to acquire movies. You can pick movies you want as soon as they are released. They may not be avaiable for months after they are released into theaters, but you don’t have to care. As soon as a version becomes available on the net, couchpotato will download it.