Nick Coleman: internet

I Moved From Google-chrome To Firefox

Nick — Thu, 10 May 2012 11:49:00 -0700

Several weeks ago, I changed my default browser from Google-chrome to Firefox. Here's why.

I had been using Google-chrome for over a year and had been a pretty happy user. I liked the Google-chrome user interface and Google had clearly put a lot of thought into how a user interacts with their browser. Neat touches were everywhere.

I did have some niggling privacy concerns, after all Google-chrome is created by the biggest data mining company in the world, but I figured Google had too much to lose by ignoring privacy in a browser. That is especially so for a browser whose source code is available to all.

Then I discovered that Google-chrome forbids the Ghostery add-on from blocking the Doubleclick network. Ghostery is a popular add-on that prevents third-party sites from monitoring your browsing, and Doubleclick is a monitoring and advertising network owned by Google.

Apparently, Google has no problem with bending the privacy rules to advantage itself.¹ Time to give Firefox a go.

I hadn't used Firefox seriously for years and was pleasantly surprised. It starts fairly quickly and is not a huge memory pig like Google-chrome. It is a little plain out of the box, but some add-ons soon fixed that.

A few months later, this is what I am running. For privacy:

adblock-plus, to block ads
ghostery, to block third-party cookies and scripts
noscript, to block any javascript
requestpolicy, to block any third-party reference
refcontrol, to control the referer header so that my browsing history doesn't leak to third parties, and
cookiemonster, to control cookies.

Then I wanted to enhance Firefox to give a good user experience like Google-chrome. These plugins did the trick:

add to searchbar, add new search engines to the search toolbar
customizable shortcuts, create your own keystroke shortcuts
duplicate in tab context menu, duplicate a tab
firegestures, similar to vimium but light-weight, uses Vim keystrokes to control the browser,
open link in ..., adds extra options to the right-click context menu, and
sessionmanager, save and load sessions by name, and keeps a history.

I now have the same facilities in Firefox that I had in Google-chrome, plus it uses about a third the memory and addresses all my privacy niggles.

(I've used the exact name of the add-ons above so you can search for them easily.)

[1] I may be attributing malice to something that is merely a by-product of how Google-chrome handles plug-ins. Google-chrome apparently does not guarantee the order that various plug-ins see the DOM, which means that other plug-ins may have applied their magic before Ghostery, thereby removing any Doubleclick references before Ghostery sees it. Occam's razor probably applies. ↑

iiNet Wins High Court Appeal by Movie Studios

Nick — Fri, 20 Apr 2012 08:44:00 -0700

In great news for anyone opposed to the heavy handed actions of the movie industry in attempting to prevent piracy, the High Court has ruled that iiNet has no direct power to prevent its users from downloading pirated content.

The music and movie industry had previously sued iiNet for, in effect, "authorising" illegal downloads by not preventing its users from doing so. They lost that case, appealed, and lost the appeal. They then appealed to the High Court. They have lost that and the matter is closed.

Until the government changes the law. Which they will undoubtedly do, since they are about to sign the TPP without any public discussion. The U.S. has previously bullied trade partners into changing their copyright enforcement laws to favour the U.S. movie industry. There is no doubt in my mind that they will do this again in the latest TPP round. The fact that the government is super quiet on the TPP makes me think they already know that they are going to have to sell out Australia's citizens.

The High Court decision does not mean that an ISP is immune, however. One part of the Federal Court appeal's result was that it provided a set process whereby an ISP could be held liable. The movie industry lost the appeal because the process they followed did not lead to iiNet's liability.

Nevertheless, for the moment, we are protected from such silliness as losing our internet connection or being sued for thousands of dollars because our teenager downloaded one stupid song (that we hate because everyone knows no-one has written decent music since the '80s, right?).

The Australian has a good summary of the decision and its history. My previous posts on this topic are here and especially here.

For the record again, my position is the common one:

I don't condone piracy, but the studios have no right to force a third party to give them my personal details. Let's not forget the studios are private corporations. They are not the police. If they want my details, a court order will give it to them. Of course, the burden of proof then falls on them, a complication they do not want to deal with despite any concepts of fairness or due process.

Google Search Verbatim Instead Of + (plus)

Nick — Wed, 28 Mar 2012 12:01:00 -0700

You probably know that Google turned off the + modifier for searches. A right royal pain in the backside for those of us who know exactly how to search and don't want the search engine to try and second guess us — it inevitably gets it wrong and returns a bunch of results we are not interested in.

The word from Google at the time was that we could use "..." instead and it would work. Well, it didn't and doesn't: Google still corrects words and makes substitutions within the quotes and defeats our attempt to narrow down the search.

It turns out that Google has added a new search parameter: verbatim. Select More search tools −> Verbatim from the menu at the left. It is a two step process instead of just being able to use + in front of a keyword, but it is better than before.

How To Set Up A Light-Weight On-line Thesaurus For Vim Pt.II

Nick — Sat, 25 Feb 2012 09:00:00 -0700

Vim has support for a built-in thesaurus. However, it consumes memory and its auto-complete selection has issues. In Part I, I showed how to set up an on-line thesaurus. Here is how to build syntax rules that will colour the output.

Summary

This is the second post of two about a light weight way to implement a thesaurus. In Part I, I described how to set up a script that provides access to an on-line thesaurus. In this Part, I describe how to write a set of simple syntax rules to provide colour and highlighting for the output.

Here is a screenshot of the finished syntax rules (using dummy data):

If you don't like these colours, you can change the rules to use whatever colours you prefer.

How Vim does it

Vim does syntax highlighting in two parts:

A set of rules that define a (group of) word, and
A colour description for each rule.

Set of rules

The set of rules is in a file named for the filetype in either $VIMRUNTIME/syntax or your ~/.vim/syntax. Each rule defines a name for that rule and a list or a regular expression to match words that should be captured by that rule.

An example is

syntax keyword cStatement goto break return continue asm syntax keyword cLabel case default syntax match cTodo /\/* *TODO/ syntax region cBlock start="{" end="}"

The first and second lines are the simplest case of keyword, which is a list of words. The third line is the next simplest case of a regex pattern match, in this case matching "TODO" within a C-style comment leading with "/*". The fourth line is like a broader match in that it specifies a region defined by a start pattern and an end pattern.

Notice each rule has a name that is a unique identifier: cStatement, cLabel, cTodo and cBlock.

In fact, it can be much more complicated than that. Rules can be contained within rules (for example, nested if-thens), rules can apply only if another rule has triggered (a '(' as part of a condition, but not as part of a comment), rules can apply to entire regions, and so on.

However, for our simple purpose which is to provide slight colour hints to a plain text list, we can ignore all that, to which we say, "Thank goodness." Check out sh.vim or even simple old c.vim to see why.

Colour description

The colour description is usually in the colorscheme you are using. For each rule name, it provides a colour scheme to use, along the lines of

hi Comment term=bold ctermfg=8 guifg=#7C7C7C

which translates as: for rule name "Comment", for a simple terminal set to bold, for a colour terminal set the foreground to colour number 8, for the GUI version of Vim set the foreground to colour number #7C7C7C.

You can think of the process as going like this: when Vim displays a file, each token or word is checked against the set of rules and assigned a rule name, and is then colourised according to the highlighting for that rule name.

You can see your highlighting description with :hi Comment, which shows the current highlighting for rule "Comment" for the filetype in the current buffer. To see all highlighting, use :hi by itself.

From this you can see that the rules and the highlighting scheme are tightly coupled. Each rule has a name, and that name should be in the colorscheme¹ . Vim controls the tight coupling by providing a set of standard names which, if we were writing rules for a new programming language, we should use. Those names are things like "Comment", "Keyword", "Statement", and so on.

However, we are creating our own set of rules for this specific instance of text, so we can do what we like. Which we shall.

Syntax rules

The data

Here is the (made up) sample text we will be working with:

Main Entry: which pronunciation [hwich, wich] [IMG] Show IPA/wItS, Definition: what Synonyms: and that, that, whatever, whichever in order that, in that, so, so that Notes: in current usage, that refers to persons or things and which is used chiefly for things. The standard rule says that one uses that only to introduce a restrictive or Antonyms: none

Main Entry: this Definition: the one Synonyms: that, the aforementioned one, the one in question, the thing indicated, this one, this person

Typically, you get a set of entries comprising Main Entry, Definition, Synonyms, Notes and Antonyms. Not every entry is present, often Antonyms and Notes are absent.

As well, if the word is common, you often get repeating sets of entries for very similar words, as in the example where the entry for this is also shown. (This isn't real data, by the way; this doesn't appear with which normally.)

Set up some rules

How to define some rules? Some things come to mind immediately.

Entry names distinguished (highlighted differently) from entry contents, i.e. "Main Entry:" should be different from "which".
The main entry word should be bolded or similar to make it stand out.
The main word melds too easily into "pronunciation..", which makes it hard to pick out. Diminish everything to the right of the main word.
Each entry should be distinguished so you can scan them easily. For example, "Main Entry" should stand out, "Definition" less so, "Synonyms" more so.

Here is an example rule set that roughly implements the first, second and fourth items.

" Include the colon as part of the word
setlocal iskeyword+=:

" Rules
" a keyword
syntax keyword  thesSynonyms Synonyms:
" this entry name has a space, so needs a regex 
syntax match  thesMainEntry /Main Entry: */ 
" this entry should include the line to the end
syntax region thesDefinition start=/Definition: /  end=/$/

" Highlighting
" link the highlighting to the defined 
" name "Keyword" in the colorscheme
hi link thesMainEntry Keyword
hi link thesSynonyms  Statement
hi link thesDefinition  Todo
" specify the colours directly
hi  thesMainEntry term=bold 
 \ctermfg=White cterm=bold guifg=6 gui=bold
" or, keep the current colour, but bold it
hi  thesMainEntry term=bold cterm=bold gui=bold

which gives this:

You can see that the keyword "Synonyms:" has matched a rule set, and the highlighting for the rule "Statement" in my colorscheme (likely different to your colorscheme) has been applied.

The regex pattern for "Main Entry: " has matched a rule set and the highlighting for the rule set has been applied.

Similarly for "Definition", the rule set has matched to the end of the line, and the highlighting for the rule "Todo" in my colorscheme has been applied.

The real deal

Rules

Enough examples. Here is the actual syntax file to produce the highlighting in the screenshot at the top of the page.

" Vi syntax file
" Language: text dump from online thesaurus
" Maintainer: Nick Coleman
" Last Change:  2012 Feb 18
" Remark: for the online thesaurus script by Nick Coleman
  
if exists("b:current_syntax")
    finish
endif

" Setup
" syntax clear    " only useful for testing
syntax case match
setlocal iskeyword+=:

" Entry name rules
syntax match thesMainEntry /Main Entry: */ contained
syntax region  thesDefinition start=/Definition: /  end=/$/
syntax keyword thesNotes Notes: contained
syntax keyword thesSynonyms Synonyms:
syntax region  thesAntonyms start=/Antonyms:/  end=/$/ 

" give the pronunciation region a special name
syntax region thesPronunciation start=/pronunciation \[/ end=/$/ contained

" Entry contents rules
syntax region thesMainWord start=/Main Entry:/  end=/$/ contains=CONTAINED keepend
syntax region thesNotesEntry start=/Notes:/  end=/^ *$/ contains=thesNotes,thesAntonyms keepend 

" Highlighting

hi link thesMainEntry     Keyword
hi  thesMainWord      term=bold cterm=bold gui=bold
hi link thesDefinition      String
hi link thesNotes     Number
hi link thesNotesEntry      Number
hi link thesSynonyms      Statement
hi link thesAntonyms      Todo
hi link thesPronunciation   Comment

let b:current_syntax = "thesaurus"

Some rules have "contains". This allows a rule within a rule, the classic example being Todo appearing within a comment where you want a different colour to make Todo stand out. "keepend" is part of that, it stops both rules at the first end pattern match rather than the final end match. :h usr_44 section 44.5 for more.

Setup

Recall from Part I that the script sets the thesaurus' buffer to filetype thesaurus. Put the above syntax file in $HOME/.vim/syntax/thesaurus.vim and the buffer will pick up the syntax rules automatically.

Windows users can put it in C:\Program Files\Vim\vimfiles\syntax\thesaurus.vim or the equivalent if using Vista or Windows 7. If you don't have administrator privileges, find where Vim thinks $HOME is by (within Vim) trying :echo $HOME or :version and putting it in $HOME\vimfiles\syntax\thesaurus.vim.

Trying it out

You probably want to use your own highlighting. A tip: to easily see the effect of your changes reload the highlighting for the data buffer with :setlocal filetype=thesaurus.

To see the colours that your colorscheme uses for a particular rule use :hi as in :hi Statement. To see all colours, use hi by itself :hi.

[1] I said the rule should be in the colorscheme. In fact, it is not an error if there is no colour highlighting for that rule, it simply gets ignored. And you can put highlighting descriptions anywhere, such as your .vimrc. For example, I quite like the colorscheme I use, except for the Search colours which I override with a separate description in my .vimrc like this:


hi Search ctermfg=white ctermbg=darkblue

. ↑

How To Set Up A Light-Weight On-line Thesaurus For Vim

Nick — Fri, 17 Feb 2012 08:02:00 -0700

Vim has support for a built-in thesaurus. However, it consumes a lot of memory, which you may not want for a feature you do not use much, and its auto-complete selection has issues. Here is how to set up an on-line thesaurus query that is light weight.

Summary

This is the first post of two (second here) about a light weight way to implement a thesaurus. It is great for what I need, which is the occasional use of a thesaurus for writing text such as this article. Once it is set up, you can forget about it and just use K whenever you want to look up a word.

A nice bonus or synergy of using an online source is that the website also returns a definition for the word, so it functions as a simple dictionary as well.

The second post (here) will deal with how to use Vim's built in syntax rule sets to provide highlighting and nice colours.

There are two or three simple steps:

a vim script that passes the cursor word to an external shell script.
a shell script that looks up the word using an online thesaurus, then parses the output to remove unnecessary cruft.
an optional syntax file to provide highlighting and colours.

Here is what it looks like with a quick-n-dirty syntax rule set:

Sample output for "quick"

I use Vim for programming and it is great for that. I use it when I need to edit files on remote servers that I have ssh'ed in. I also use Vim for writing text because it is a great text editor. I am writing this in Vim, for example.

Often, one of the things you want when writing text is a thesaurus, which is the topic for today.

Why on-line

Vim comes with support for a thesaurus, but I've never really liked it, for two reasons.

You download a thesaurus (the Moby one is common), point Vim to it and you are good to go. However, a thesaurus can consume a lot of memory. The Moby file is 24 MB, although to be fair Vim doesn't use anything like that. If you are using Vim remotely, you may not want to use any extra memory. For example, I have a remote VPS that has only 128 MB of total memory, so every megabyte is critical. This is especially true if you use the thesaurus only rarely; the trade-off is not worth it.

The bigger issue for me is that Vim does not handle a long list of alternative words very well. It is fine with a short list where you use it just like auto-complete, but it seems to get confused with a long list. Unfortunately, Moby often serves up a long list, and you frequently find you have inadvertently changed your perfectly good word to something completely different. You then have to Undo and try again.

In fact, I got so frustrated with the Vim/Moby combination I ended up unsetting the thesaurus feature in Vim after just a couple of months.

The good news is that there is an alternative, and that is to use one of the many on-line thesauruses.

I have set things up so that a script calls a command-line browser to query for a word and dump the output from the online website, then parses it to remove extra cruft such as sidebars, headers and footers, and white space, and puts it into a scratch buffer in Vim itself for me to browse through and perhaps copy a word from.

Vim setup

First, set up Vim to call a script, which I will describe further down, and display the results in a scratch buffer. (I shamelessly pulled the concept straight from the ReadMan script, which displays a Unix man page for the keyword over the cursor.) To use it, simply press K with the cursor under the word that you want to see alternatives. You can do all the normal things in the buffer like search, jump to a line, copy a word (cw), and so on. To close it, just hit the q key.

Copy and paste the following into your .vimrc or wherever you prefer. I have it in a common.vim file in my ftplugin directory, where my various filetype scripts can source it if they want.

fun! ReadThesaurus()
   " Assign current word under cursor to a script variable
   let s:thes_word = expand('')
   " Open a new window, keep the alternate so this doesn't clobber it. 
   keepalt split thes_
   " Show cursor word in status line
   exe "setlocal statusline=" . s:thes_word
   " Set buffer options for scratch buffer
   setlocal noswapfile nobuflisted nowrap nospell 
     \buftype=nofile bufhidden=hide 
   " Delete existing content
   1,$d
   " Run the thesaurus script
   exe ":0r !/home/nickcoleman/bin/thesaurus " . s:thes_word 
   " Goto first line
   1
   " Set file type to 'thesaurus'
   set filetype=thesaurus
   " Map q to quit without confirm
   nmap <buffer> q :q<CR>
endfun
" Map the K key to the ReadThesaurus function
noremap <buffer> K :call ReadThesaurus()<CR><CR>

The script is fully commented for you to follow what is going on.

Notice I specified the location of the shell script that Vim is calling to be $HOME/bin/thesaurus. Change this to wherever you are going to put the shell script.

I call the scratch buffer "thes_" so that I have a name for it which means I can easily re-use the buffer again. Otherwise Vim would create a new buffer every time and go through buffer numbers like crazy. In the unlikely event you have an actual file called "thes_", change the script to use some other wacky name. I could have used the Vim function tempname() to generate a unique buffer name, but "thes_" is meaningful and easy to remember if you want to unhide the buffer later.

I set the status line to show the word I am looking up. Sometimes the website will return a different word, a near synonym, so the status line reminds me exactly which word I am looking up. An example is looking up the word "the" which returns the definition and synonyms for "histrionical". No, I don't know why.

I include a line to set the filetype to "thesaurus". Its purpose is to allow me to set up anything special for that filetype later on. A possible use would be to create a syntax file to do special highlighting or colouring. That will be the topic for the next post.

The mapping for K has a second . It eats up the "Press ENTER or type command to continue" prompt. There are other ways to do this, but they can mess up the display or have the side effect of applying globally instead of just this buffer.

By the way, in case it isn't obvious, the reason I split the functionality in to two—part- Vim and part-shell scripts—instead of doing it all in Vim is because Vim's scripting language, always an awkward beast at the best of times, would make it too hard. Best to use the good parts of each and combine them.

Thesaurus script

Up to now, everything has been operating system agnostic. The script below is written for unix-like operating systems, including OS X, because it uses some unix utilities like links and sed. (The others it uses such as basename and readlink aren't absolutely necessary, just good style.) I have written a paragraph or two below on how to get those tools for Windows.

Originally I wrote a quick-n-dirty one-liner, but I decided the script might be useful outside Vim as well, so I tidied it up to be a useful script that you can call from the command line.

This is the shell script that Vim calls. Put it in the location you specified in the Vim script above. It is straightforward, apart from the sed call, and there are enough comments so you can see what is happening.

#!/bin/sh

# This searchs an online thesaurus, cuts as much cruft out as possible,
# and displays the definitions.

#URL='http://www.merriam-webster.com/thesaurus/'
URL='http://thesaurus.com/browse/'

# Display help, and quit.
function usage {
    cat << EOF_HELP
    Usage: $(basename $0) [-r][-h] word
    Display the parsed results of an online thesaurus search for 

    -r  Raw results; no filtering.
    -h  Display this help.

EOF_HELP
    exit 
}

# Check for a parameter.  Get the output from the website
function get_thes {
    [  -z "$1" ] && usage
    $(readlink -e $(which links)) -dump "${URL}$1" 
}

# Check for a -h parameter or absence of parameter.
[ "$1" = "-h" -o -z "$1" ] && usage

# If -r just get the raw output, otherwise pass it through sed.
if [ "$1" = "-r" ] ; then
    shift
    get_thes $1
else
    # Sed is doing many things: 
    # Print out "no results found for " and quit. 
    # Print out only the lines I'm interested in, which are:
    #   No results found for
    #   Main Entry:
    #   Definition:
    #   and all from Synonyms: to Antonyms:
    # Put a new line after Antonyms.
    get_thes $1 | sed -n -e '/No results found for/ {' \
      -e ' s|^[ \t]*\(No results .*\)|\1|p ; q} '\
      -e '{/Main Entry:/p ' \
      -e '/Definition:/p ' \
      -e '/Synonyms:/,/\(Antonyms:\)\|\(^$\)/p} ' \
      -e '/Antonyms:/ a \ '
fi

I use thesaurus.com because I found it has a wider coverage than merriam-webster.com. If you want to use a different website, you will need to write your own sed script to parse it.

I separated the sed actions into discrete chunks above so you can get a clearer picture of what sed is doing, but they appear as one line in the actual script. That line is below, for you to cut-n-paste.

    get_thes $1 | sed -n -e '/No results found for/ { s|^[ \t]*\(No results .*\)|\1|p ; q} ;{/Main Entry:/p ; /Definition:/p ; /Synonyms:/,/\(Antonyms:\)\|\(^$\)/p} ; /Antonyms:/ a \  '

Install

I prefer that this script is only available in buffers where I am writing text. That way, I can keep the K mapping for unix man pages in buffers where I am writing code.

The way to do that is to put the Vim script in its own file that is called only by text-like filetypes. Scripts that are unique to filetypes are put in ~/.vim/ftplugin or ~/.vim/after/ftplugin. I created files named text.vim, xml.vim and blog.vim that are in the ftplugin directory. (XML is treated as text because most of my XML is text data.) I put the thesaurus script in a file called common.vim and I source common.vim from within all the above ftplugin scripts.

If ftplugin (and perhaps autocmd) are a mystery, see :h ftplugin and :h autocmd.

With that, I am done. To use, move the cursor in Normal mode underneath the word of interest and press K.

Bugs

I noticed once that the script loses permissions to a temporary file that Vim uses internally. It seems to only happen if a remote ssh session in screen is unexpectedly terminated, and not always then. Closing and re-opening Vim (yes, a pain) will fix it.

Additional Info for Windows Users

Summary

In summary, Windows users have a little extra to do, but not much.

Install sed and links. It takes only a few seconds each to download and install. The default install is fine.
Create a batch file to get and parse the thesaurus data. The batch file will be an abbreviated form of the shell script above, tailored for Windows.
Point vim to the location of the batch file.

Install Links & Sed

Windows users will need to install the links and sed commands and perhaps a shell if you don't want to use a DOS batch file (which is below). I did a bit of a search and found a Windows build of links here, and a good set of unix tools for Windows here (direct link for sed is here), both of which are fine.

Create DOS Batch File

You don't need the full-blown shell script above. The simple DOS batch file below will do. It runs links and then pipes the output to sed. You might need to change the paths to the folder(s) where you installed links and sed if you did not use the defaults when you installed them.

@"c:\program files\links\links.exe" -dump http://thesaurus.com/browse/%1 | "c:\program files\gnuwin32\bin\sed.exe" -n -e "/No results found for/ { s|^[ \t]*\(No results .*\)|\1|p ; q} ;{/Main Entry:/p ; /Definition:/p ; /Synonyms:/,/\(Antonyms:\)\|\(^$\)/p} ; /Antonyms:/ a \  "

There are a couple of differences to the unix script. Sed needs double quotes surrounding its commands instead of single quotes. The paths to the executables need double quotes because of the spaces in the path. I put a '@' in front of everything to prevent DOS from echoing back the entire command. Finally, I put %1 instead of $1 for the DOS way to pass in the parameter.

It is probably worth opening a cmd.exe window and testing the batch file. Assuming you called the batch file "thesaurus.bat", run thesaurus.bat loose and you should get a listing back after a few seconds with all the synonyms of "loose".

Point Vim

Now that you have a batch file that works, put it in a folder somewhere. The vim script needs one small change to point to that location.

In Windows, $HOME expands to "C:\Documents and Settings\{user}".¹ However, it doesn't expand in a script if it is contained within quotes. So the vim script should be changed from

" Run the thesaurus script :exe ":0r !$HOME/bin/thesaurus " . s:thes_word

" Run the thesaurus script :exe ':0r !"' . $HOME . '\thesaurus.bat" ' . s:thes_word

assuming again that your DOS batch file is called "thesaurus.bat". This puts $HOME outside the quotes and Vim will expand it to "C:\Documents and Settings\{user}\thesaurus.bat"¹.

If you put the batch file in a sub folder, add that folder name in front of '\thesaurus.bat' like this: '\my_folder\thesaurus.bat'.

If it does not work, because you tested the batch file itself beforehand, the problem is almost certainly in how you specified the location of your batch file in the vim script.

[1] for Windows XP. Vista expands to "C:\Users\{user}". Windows 7 the same as Vista? -- I don't know, I don't have it. ↑

Get Google-chrome's Version (2)

Nick — Mon, 21 Nov 2011 10:27:00 -0700

I wrote previously about a small script to extract the Google-chrome's version number from its deb file. To recap, Google provides either a deb (a deb file is a Debian package) or an rpm (for Redhat and its kin, such as Ubuntu) file only. Other unixes and Linux distributions need to create their own packages, and Slackware provides a script to pull the deb file apart and repackage it into a Slackware package.

I wrote a small script based on the Slackware script that extracts the google-chrome version number from the deb file so you can see if you have the latest version or not. (None of this would be necessary if Google would name their deb file to include the version number. I don't know why they don't do this.)

I noticed the script is not particularly efficient so I rewrote the section that actually gets the version number. Here it is (first the original, then the update):

 # The original
VERSION=$(ar p "$FILE" control.tar.gz 2> /dev/null | tar zxO ./control 2> /dev/null \
	| grep Version | awk '{print $2}' | cut -d- -f1)
 # A better version that does everything in awk
VERSION=$(ar p "$FILE" control.tar.gz 2> /dev/null | tar zxO ./control 2> /dev/null \
	| awk -F "[ :-]" '/Version/ {print $3}')

The difference is that the second version uses awk for everything instead of using grep then awk then cut, so it avoids two extra pipes and sub-shells.

It is small beer, but, since some of these scripts are by their nature tutorials, we should probably use the better versions. Without meaning to disparage the original authors, using grep then awk is a classic beginner's mistake and shouldn't serve as an example of best practice.

Details

This is where I go into detail. Skip it if you are not interested.

ar p "$FILE" control.tar.gz 2> /dev/null | 
	tar zxO ./control 2> /dev/null

produces the following output:

Package: google-chrome-stable Version: 15.0.874.121-r109964 Architecture: i386 Maintainer: Chrome Linux Team Installed-Size: 104068 Pre-Depends: dpkg (>= 1.14.0) Depends: libasound2 (>> 1.0.22), libbz2-1.0, libc6 (>= 2.11), libcairo2 (>= 1.6.0), ... ... ... Description: The web browser from Google Google Chrome is a browser that combines a minimal design with sophisticated technology to make the web faster, safer, and easier.

The line we are interested in is the second line, Version: 15.0.874.121-r109964. We don't need to grep for that line, awk can get it itself with /Version/.

We also don't need to cut out the version number, we can separate the line into fields in awk and print out the relevant field. Looking at the line, the field we want is preceded by a space and succeeded by a dash. We use the command line option -F "[ –]" which gives awk a regular expression to use as field separators. That is, anything separated by a space or a dash is a field. Then print the second field ({print $2}).

The result of this is 15.0.874.121.

Get Google-Chrome's Version from Deb File Download

Nick — Wed, 09 Nov 2011 08:08:00 -0700

Here is a quick script to pull out google-chrome's version number from the deb file that you download from Google's website. I call it getversion.sh. I find it useful when you want to see if the google-chrome version is newer than the one you have installed on your Slackware system without having to go through the whole build process.

The method is pulled from Slackware's build script which itself thanks Fred Richards.

To use it: ./getversion.sh [ -h | filename ] If you don't provide a filename, it will look in the current directory for the default-named stable deb file. It will print this: google-chrome-stable_current_i386.deb file's version is: 15.0.874.106.

Because the output is separated by a ":", in other words into tokens, you can use it in a script to get just the version number. ./getversion.sh | awk -F: '{print $2} will output just the number like this: 15.0.874.106.

You could put that functionality within the script and signal it with a command line flag, but I haven't bothered. I leave it as an exercise for the reader.

Here is the script:

#!/bin/sh
if [ "x$1" = x-h ] ; then
    cat <<-END_HELP
    This extracts the version number of google-chrome from the deb file specified on the command line
    or in the current directory if none is specified. (Cobbled from the Slackbuild script.)
END_HELP
    exit
fi
RELEASE=${RELEASE:-stable}    # stable, beta, or unstable
case "$(uname -m)" in
  i?86) DEBARCH="i386" ; LIBDIRSUFFIX="" ; ARCH=i386 ;;
  x86_64) DEBARCH="amd64" ; LIBDIRSUFFIX="64" ; ARCH=x86_64 ;;
  *) echo "Package for $(uname -m) architecture is not available." ; exit 1 ;;
esac
FILE="google-chrome-${RELEASE}_current_${DEBARCH}.deb"
if [ $# -gt 0 ] ; then
    FILE="$1"
fi
if [ -f "$FILE" ] ; then
    # Get the version from the Debian/Ubuntu .deb (thanks to Fred Richards):
    VERSION=$(ar p $FILE control.tar.gz 2> /dev/null | tar zxO ./control 2> /dev/null |
		grep Version | awk '{print $2}' | cut -d- -f1)
    echo "$FILE file's version is: $VERSION"
else
    echo "$FILE not found."
fi

Decoding a Spammer's Attempt to Obfuscate His IP Address

Nick — Sun, 18 Sep 2011 18:47:00 -0700

IP v4 addresses are familiar as dotted quads. A spammer uses an interesting feature of IP addressing to obfuscate his address. A look at the various ways of specifying an IPv4 address.

I received a spam email yesterday that was phishing for some banking details. It contained the usual "your account has been disabled, you need to reactivate it" spiel, along with a link to click. When I hovered the cursor over the link, it displayed the bank's URL. Or did it?

URL displayed when hovering over the link

A cursory glance shows the link's URL pointing to firstdirect.com. There is some other stuff before the firstdirect.com, but I wonder if many people would query it, especially since it appears to contain the quite-common www1 prefix.

If we look closely, however, we can see that the actual domain part of the URL is 95.11064393, which is followed by a directory of www1.firstdirect.com. There is a "/" almost hidden between the two parts. It is quite easy to overlook, which is probably the phisher's intent.

We all know this is a common technique of phishers: make the bank name appear as though it is part of the domain name. The bit that I found interesting was the domain. It is not a fully-qualified domain name; it doesn't have a .com, .org, etc, so it must be an IP address. But it is not in the familiar dotted quad notation of 123.123.123.123. How does it work?

IP Addresses Revisited

An IP address is simply a number, a 32 bit number. It can be represented in dotted-decimal quad notation, the most common, or it can be in decimal, hexadecimal, octal or even binary. For dotted quad, the four component parts are just the number × 256ⁿ, where n is 3 for the left-most part and 0 for the right-most part.

Therefore, for IP address 74.125.237.20 (a Google server), the decimal equivalent is

74×256³ + 125×256² + 237×256¹ + 20×256⁰ = 1249766676

and we can test that by pinging it:

$ ping 1249766676 PING 1249766676 (74.125.237.20) 56(84) bytes of data. 64 bytes from 74.125.237.20: icmp_req=1 ttl=53 time=65.1 ms ^C

Here we see that pinging 1249766676 tells us two things:

it is a valid address since the server does respond;
an IP address works for decimal addresses as well as dotted quad addresses.

It also works for hexadecimal. 1249766676 is 0x4a7ded14 in hexadecimal:

$ ping 0x4a7ded14 PING 0x4A7DED14 (74.125.237.20) 56(84) bytes of data. 64 bytes from 74.125.237.20: icmp_req=1 ttl=53 time=63.6 ms ^C

and it works.

Interesting

Now it gets interesting. Up to now, we have converted numbers in their entirety. You can also convert them and still use dotted quad notation. So you can convert 74.125.237.20 to dotted quad hex, which is 0x4a.0x7d.0xed.0x14 and it works:

$ ping 0x4a.0x7d.0xed.0x14 PING 0x4A.0x7D.0xED.0x14 (74.125.237.20) 56(84) bytes of data. 64 bytes from 74.125.237.20: icmp_req=1 ttl=53 time=61.9 ms

and you can even mix them up with some quads in decimal and some in hexadecimal:

$ ping 0x4a.125.0xed.20 PING 0x4A.125.0xED.20 (74.125.237.20) 56(84) bytes of data. 64 bytes from 74.125.237.20: icmp_req=1 ttl=53 time=62.5 ms

More Interesting

To recap, you can use a variety of number bases such as 2, 8, 10 and 16 to specify an IP address, and you can do that with the entire number or with each dotted quad separately. Now we get to the really interesting bit.

You can use part dotted quad and part whole number. This is what the phisher has done. The IP address of 95.11064393 is using the first quad and then an entire number for the second, third and fourth quads. It translates to:

95×256³ + 11064393 = 1604899913 $ ping 1604899913 PING 1604899913 (95.168.212.73) 56(84) bytes of data. 64 bytes from 95.168.212.73: icmp_req=1 ttl=52 time=299 ms ^C

The spammers IP address of 95.11064393 translates to 95.168.212.73. Who is that?

$ whois 95.168.212.73 inetnum: 95.168.212.64 - 95.168.212.127 netname: SUPERNETWORK-STUDIO51-3 descr: Studio51 s.r.o. country: CZ

is a server in the Czech Republic.

For those who want the math of converting the number to dotted quad, it is just taking successive integer division of decreasing powers of 256. It sounds more complicated than it is, so here is the bc¹ script (scale in bc sets the number of decimal points; we want whole numbers so set it to 0):

$ echo 'scale=0; a=11064393; x=a/256^2; b=a%256^2; y=b/256; z=b%256; print x,".",y,".",z,"\n"' | bc 168.212.73

which you can see is the second, third and fourth quads that are shown in the ping above.

Seriously

I had planned to put a link to the various RFCs for IP addressing, but, because the IP protocol is so old and various bits have been changed many times, it gets too confusing, not least to me. Instead, here is a link to a good article on Wikipedia about IP addressing.

The main idea to take away is that IP addresses are simply a 32 bit number, and the various utilities that we use, such as ping and our browsers, are capable of accepting that number in a wide variety of formats, some them quite surprising.

As for our spammer, I guess he was clever in trying to obfuscate the URL so that it looks less like an IP address. It didn't work though.

All IP addresses in this post refer to IPv4. IPv6 is a different kettle of fish.

[1] bc is a unix calculator utility ↑

I used bc to calculate all the numbers for this post, such as this example for hexadecimal output: $ echo 'obase=16; a=74*256^3+125*256^2+237*256+20; print "0x",a,"\n"' | bc 0x4A7DED14

Prey: Replace Streamer with Mplayer

Nick — Sun, 11 Sep 2011 10:35:00 -0700

Prey is a recovery application for stolen mobile devices that, among other things, takes a photo through the webcam. It uses streamer, which can be problematic. Replace streamer with mplayer.

Prey is an application that can help you track your mobile device if it is stolen. It is pretty neat and I have it installed on my laptop.

It does its job by checking in with a central server every so often. If your device is stolen, you mark it on the server (using another computer) and then prey, upon checking in and realising the device is marked as missing, will run its recovery routine. It gathers information about the device's IP address, GPS location if available, MAC address and so on, takes a snapshot of the screen, and takes a photo through the device's camera. It bundles that up and sends it to the central server, or you can tell it to send to you in an email.

Armed with that information, you can set about recovering your device.

Prey is available for all the major platforms: Windows, Mac and Linux. It uses common unix utilities (compiled especially in the case of Windows). For Linux it assumes that those utilities are already available on the system, which for the most part they are. However, streamer, the webcam photo taker, is not. Many distributions have it as part of the xawtv package. That means you have to pull in the entire xawtv package and dependencies just for one application, streamer.

I tried building xawtv on my Slackware 13.37 machine and couldn't without having to jump through hoops, which I didn't feel like doing just to get one application out of it. Streamer itself built ok, but it couldn't find the webcam to grab. I don't know if that was because of the build problems or was something else, but I was not inclined to investigate because there is an alternative.

Every major Linux distribution has access to an application that is quite capable of taking photos: mplayer. I decided to see how easy it would be change streamer to mplayer.

Let's find out where streamer is used.

prey $ grep -r -m 1 streamer * README: o streamer (for webcam capture) --> in Fedora the xawtv package includes it modules/webcam/config:# frames per second (for streamer in linux) modules/webcam/platform/linux/functions: # do we have streamer installed ?

That is good: streamer is in the README, a config file, and only in one source file. That is encouraging; this should be doable, time to see how easy it is to take photos with mplayer.

I've done this before so I know the options for mplayer:

mplayer -tv driver=v4l2:fps=2 -vo jpeg -vf framestep=10 -frames 10 tv://

From left to right, this sets up the streaming video capture module "tv", uses the webcam driver "v4l2" (a common one for many webcams) and sets it to 2 frames/second, sets the output to jpeg, renders every 10 frames which is 5 seconds worth, saves 10 frames, and uses the tv: device.

I use framestep to give the webcam time to warm up. Many cams need a second or two to get the brightness and contrast levels correct and they futz around during that time. We need the camera to be switched on for at least a couple of seconds, and we take 10 snapshots and use one of the later ones.

Now back to prey. I know from the grep above that streamer is used in file modules/webcam/platform/linux/functions. Here is a listing:

#!/bin/bash #################################################################### # Prey Webcam Module Linux Functions - by Tomas Pollak (bootlog.org) # URL: http://preyproject.com # License: GPLv3 ####################################################################

take_picture() { # do we have streamer installed ? local streamer=`which streamer` if [ -n "$streamer" ]; then # take four pictures every 0.5 seconds as JPEG $streamer -t 4 -r 0.5 -o "$tmpdir/streamer0.jpeg" &> /dev/null if [ -f "$tmpdir/streamer3.jpeg" ]; then # we got it mv "$tmpdir/streamer3.jpeg" "$webcam__picture" > /dev/null rm -f "$tmpdir/streamer{0,1,2}.jpeg" 2> /dev/null else # some webcams are unable to take JPGs so we try to grab a PPM $streamer -t 4 -r 0.5 -o "$tmpdir/streamer0.ppm" &> /dev/null if [ -f "$tmpdir/streamer3.ppm" ]; then # good

local convert=`which convert` if [ -n "$convert" ]; then # lets convert it to jpg $convert "$tmpdir/streamer3.ppm" "$webcam__picture" > /dev/null else # lets just send it as a PPM log " -- Could't find Imagemagick! Sending image as PPM." webcam__picture="$tmpdir/streamer3.ppm" fi

rm -f "$tmpdir/streamer{0,1,2}.ppm" fi fi fi }

capture_video() { # we should already know if we do have streamer if [ -n "$streamer" ]; then local frames=$(( $webcam__video_capture_time * $webcam__frames_per_second )) $streamer -o "$webcam__video" -f yuv2 -F stereo -r $webcam__frames_per_second -t $frames fi }

We can ignore function capture_video as it is not currently used in the default configuration. Looking at function take_picture, we can see that the streamer application is set in a local variable, which is used throughout the rest of the script. That makes it easy to substitute mplayer. The other thing to change is the command line options and the names of the jpegs. We also can ignore the else clause because mplayer is capable of taking jpegs so that clause will never be entered.

Here are the changes in the form of a diff:

- local streamer=`which streamer` + local streamer=`which mplayer` if [ -n "$streamer" ]; then # take four pictures every 0.5 seconds as JPEG - $streamer -t 4 -r 0.5 -o "$tmpdir/streamer0.jpeg" &> /dev/null + $streamer -tv driver=v4l2:fps=2 -vo jpeg:outdir=$tmpdir -vf framestep=10 -frames 10 tv:// 1>/dev/null 2>&1 - if [ -f "$tmpdir/streamer3.jpeg" ]; then # we got it - mv "$tmpdir/streamer3.jpeg" "$webcam__picture" > /dev/null - rm -f "$tmpdir/streamer{0,1,2}.jpeg" 2> /dev/null + if [ -f "$tmpdir/00000005.jpg" ]; then # we got it + mv "$tmpdir/00000005.jpg" "$webcam__picture" > /dev/null + rm -f "$tmpdir/000000??.jpg" 2> /dev/null else # some webcams are unable to take JPGs so we try to grab a PPM

The changes are: change the application to mplayer, change the application's command line options, change the names of the image files to the ones that mplayer generates.

I am using the 5th jpeg because, on my webcam, it is usually the first one where the camera has stabilised. You might experiment with your camera to check. You probably don't want to use the last image, number 10, in case the user/thief notices the camera light come on and covers up the camera.

I've truncated the code listings above to make them easier to read here; as a result the diff above is not a real one. You can download the actual patch and apply it. Cd to prey/modules/webcam/platform/linux, move the patch file here, and apply it with patch -p1 < prey_mplayer.patch

I hope you find this useful. With any luck, you will never need it.

Australia Debate on Cyber Data Retention

Nick — Tue, 30 Aug 2011 10:15:00 -0700

Australia is "debating" changes to the law that will force ISPs and telecommunication companies to keep data on their users. I put "debating" in quotes, because the government is forcing the bills through without any sort of meaningful public input.

The bills are a disgrace. As one commentator (see link below) says, we don't accept that the government can open our letters and read them, so why should email or text messages be any different? We also haven't been told whether a history of our day-to-day browsing the Web will be kept, or for how long, or who would have access to that information.

The trouble with broad-reaching legislation is that, despite reassuring comments at first, the legislation inevitably gets used in the widest possible way, much beyond what the original intent was. For this reason alone, we should be concerned about it.

In the interests of fairness, I link to The Age's opinion piece by Robert McLelland, the Federal Attorney-General, who is replying to a previous critical piece. Make sure you read the comments to McLelland for several very good reasons why the legislation should be rejected.