Friday, August 5, 2016

realurl

The inability of browsers to deal with shortened URLs led to the writing of a quick one-liner for verifying the final destination of a URL.

Add this line to ~/.bashrc :

alias realurl='ruby -r net/http -e "ARGV.map{ |s| while(r=Net::HTTP.get_response(URI(s)) and r.code[0] == \"3\"); s=r[\"location\"];end;puts s}"'

Then use the realurl alias to look up URLs:

bash# realurl http://goo.gl/KjdQYT
https://www.blackhat.com/us-16/briefings.html#weaponizing-data-science-for-social-engineering-automated-e2e-spear-phishing-on-twitter?Wolfesp18

The expanded Ruby code is as follows:

#!/usr/bin/env ruby
# expand a shortened URL by following all redirects
require 'net/http'

def expand_url(str)
  while(hdr=Net::HTTP.get_response(URI(str)) and hdr.code[0] == '3')
    str = hdr['location']
  end
  puts str
end

if __FILE__ == $0
  ARGV.each do |arg|
    puts expand_url arg
  end
end 

Sure, it would be nice to print the page Title attribute as well, but that means loading up Nokogiri and ends up being a bit of overkill.

Wednesday, March 23, 2016

Big-M Notation

Today's node.js chaos seemed like it was only a simple lesson in Operations 101 (hint: always host the code for your application and the modules it depends on in the same place) until one looks at the actual function that everyone was linking to:

function leftpad (str, len, ch) {
  str = String(str);

  var i = -1;

  if (!ch && ch !== 0) ch = ' ';

  len = len - str.length;

  while (++i < len) {
    str = ch + str;
  }

  return str;
}

This is quite possibly the worst algorithm in the (admittedly short) history of padding strings. Unless JS is doing something special behind our backs, we're looking at something on the order of len string allocations here, each one slightly larger than the last.

OK, so JS more or less asks for problems like this because it refuses to provide anything remotely resembling a standard library. Still, this is a common enough problem: people using dynamic memory management as if it were free, with no awareness of the costs of memory allocation and subsequent garbage collection.

To help raise awareness, I propose a Big-M notation to supplement Big-O notation (and Big-theta and all that). Big-M, or M( ), determines how memory allocations in an algorithm will grow with an input n. As with the other Big-notations, lower-order terms are dropped. 

Using this notation, the above string padding algorithm is M(n), when it should be something constant like M(1).

Wednesday, March 16, 2016

firefox profiles for ad-hoc privilege separation

Just a quick write-up, since few people seem to actually be doing this.

Firefox support profiles: each profile is a distinct configuration under ~/.mozilla/firefox, with its own extensions, themes, bookmarks, history, and cookies. This means that badly-behaved web pages in one profile will not be able to read the data from another profile (unless they actually own your system, which is beyond the scope of this simple technique). It also means you can have as many GMail accounts as you like without having to sign out and wipe cookis every time.

To create non-default profiles, open the profile manager from the command line:

bash# firefox --no-remote -P

Create as many profiles as you like. There can be one for each online persona, one for email, one for paying bills, one for pr0n, whatever. For this discussion, we will assume the profile test was created.


For each profile, do the following:

1. Create a desktop file for the profile in ~/.local/share/applications . Make sure the .desktop filename is unique.

  bash# cp /usr/share/applications/firefox.desktop ~/.local/share/applications/firefox-test.desktop

2. Add a distinct icon for the profile in ~/.local/share/icons . You can get different icons by doing a Google image search for firefox icon. There are plenty.

  bash# cp ~/downloads/firefox-green.png ~/.local/share/icons/firefox-test.png

3. Edit the .desktop file to ensure that the Name and Icon are unique, and that the Exec line references the new profile:

Name=Mozilla Firefox (testing)
Exec=firefox --no-remote -P test
Icon=firefox-test.png

That's all there is to it. A new icon for Firefox (testing) will appear in your favorite window manager. Add whatever extensions you deem fit for the purpose.

Monday, March 7, 2016

RIP inittab

Your getty configs are all in /lib/systemd/system/getty@.service now.

[Service]
# the VT is cleared by TTYVTDisallocate
ExecStart=-/sbin/agetty --noclear %I $TERM                                      
Type=idle
Restart=always
RestartSec=0
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
TTYVTDisallocate=yes
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes

Control via /etc/systemd/logind.conf -- see man 5 logind.conf for more info.

Sunday, March 6, 2016

xdg-open arcanery

In the midst of playing with tycat and terminology on a virtual console*, it became apparent that xdg-open wasn't doing its job in regards to opening PDF documents.

  bash# xdg-open /tmp/dw.pdf 
  Filename "file:///tmp/dw.pdf" does not exist or is not a regular file

A simple check of the default handler shows that everything is fine:
  bash# xdg-mime query default application/pdf
  evince.desktop
..but for fun, let's try to change it and see what happens.
  bash# xdg-mime default apvlv.desktop application/pdf
  bash# xdg-open /tmp/dw.pdf 
  Filename "file:///tmp/dw.pdf" does not exist or is not a regular file
Nothing.

Grepping for the error in /usr/bin/xdg-open comes up empty. It's odd that two unrelated programs (evince and apvlv) would have the incorrect error message (the file does exist and is regular), or that they would both fail to handle file:// URIs. How does xdg-open actually route these requests?

Turns out it depends on your window manager. If you are running Enlightenment, xdg-open performs the following:

open_enlightenment()
{
    enlightenment_open "$1"

    if [ $? -eq 0 ]; then
        exit_success
    else
        exit_failure_operation_failed
    fi
}

Replacing enlightenment_open "$1" with open_generic "$1" works. Can we invoke open_generic when enlightenment_open fails? Turns out we can't, because enlightenment_open returns 0 (success) even when the file isn't opened.

Fixing enlightenment_open might do the trick. The relevant code (enlightenment.git/ src/bin/e_open.c) is here:

        char **itr;
        int ret = EXIT_SUCCESS;

        for (itr = cmds; *itr != NULL; itr++)
          {
             /* Question: should we execute them in parallel? */
             int r = system(*itr);
             if (r < 0)
               fprintf(stderr, "ERROR: %s executing %s\n", strerror(errno),
                       *itr);
             free(*itr);
             if (r > 0) /* Question: should we stop the loop on first faiure? */
               ret = r;
          }
        free(cmds);

        return ret;

It turns out that the status of what enlightenment_open is calling is ignored. But what is it calling? Adding the line fprintf(stderr, "Trying %s\n", *itr); after the system() call uncovers what is actually happening:

bash# make src/bin/enlightenment_open
  CC       src/bin/src_bin_enlightenment_open-e_open.o
  CCLD     src/bin/enlightenment_open
bash# src/bin/enlightenment_open /tmp/dw.pdf 
Filename "file:///tmp/dw.pdf" does not exist or is not a regular file
TRYING evince-previewer 'file:///tmp/dw.pdf

Where the hell did evince-previewer come from? To make a long story short, xdg-open (and enlightenment_open, and all other *_open tools) uses /usr/bin/mimeopen as a backend. 

This utility operates on files, not mime-types:

bash# mimeopen -a /tmp/dw.pdf 
Please choose an application

1) Print Preview  (evince-previewer)
2) Document Viewer  (evince)
3) vprerex  (vprerex)
4) GNU Image Manipulation Program  (gimp)

use application #
Cancelled

OK, so that is where evince-previewer is coming from. A quick test shows that the actual bug is caused by evince-previewer not understanding file URIs:

bash# evince-previewer file:///tmp/dw.pdf
Filename "file:///tmp/dw.pdf" does not exist or is not a regular file
bash# evince file:///tmp/dw.pdf
(evince:21480): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 15

That Gtk warning accompanies evince loading and displaying the file just fine. 

Changing the handler for PDFs using mimeopen turns out to work just fine:

bash# mimeopen -d /tmp/dw.pdf 
Please choose a default application for files of type application/pdf

1) Print Preview  (evince-previewer)
2) Document Viewer  (evince)
3) vprerex  (vprerex)
4) GNU Image Manipulation Program  (gimp)
5) Other...

use application #2
Opening "/tmp/dw.pdf" with Document Viewer  (application/pdf)

(evince:21510): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 15
bash#  xdg-open /tmp/dw.pdf 

(evince:21550): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 15
xdg-open /tmp/a.pdf

(evince:21602): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to allocate widget with width -82 and height 1

Of course, that doesn't mean that evince-previewer is the only bug. Since evince-previewer returns 1 (failure), enlightenment_open should be returning 1 instead of 0:

        int ret = EXIT_FAILURE;

        for (itr = cmds; *itr != NULL; itr++)
          {
             int r = system(*itr);
             if (r < 0)
               fprintf(stderr, "ERROR: %s executing %s\n", strerror(errno),
                       *itr);
             free(*itr);
             ret =  r;
          }

...and xdg-open should check the return value and fallback to open_generic:

    enlightenment_open "$1"

    if [ $? -eq 0 ]; then
        exit_success
    else
        open_generic "$1"
        if [ $? -eq 0 ]; then
            exit_success
        else 
            exit_failure_operation_failed
        fi
    fi

Let's not even get started on why xdg-open doesn't propagate changes back to mimeopen. Wait, why doesn't it?

UPDATE: looks like there's something deeply wrong with with the xdg-* utilities:

bash# xdg-settings get default-web-browser
xdg-settings: unknown desktop environment

The problem here is that xdg-settings performs its own check for desktop environment (modularity, people!) and, of course, implements this poorly. If the desktop environment is detected (as is Enlightenment), but is not KDE|Gnome|XFCE|MATE,  it returns "unknown desktop environment". If the environment is not detected, a generic handler is called. There are some very confused people working on this software.

The fix is to modify xdg-settings, changing the default case statement
*)
    exit_failure_operation_impossible "unknown desktop environment"
    ;;

to

*)
    dispatch_specific generic "$@"
    ;;

* For those interested: install tslib and rebuild EFL from source, then run the following on a virtual terminal:
ELM_ENGINE=fb /usr/local/bin/terminology -f=nexus/20
Yes, the -b option for a backgroud image will work, as will tytls, tyop, and tycat.

Thursday, February 25, 2016

Dispatches from the X1 front

The first-gen Thinkpad X1: possibly the best laptop I've ever owned. Lasted three years, compared to the usual 12-18 months. It still runs, but the battery is down to about 40 minutes, and the (second!) keyboard is starting to lose keys.

The third-gen Thinkpad X1 Carbon: a little annoying at first: no SD slot, no h/w kill switch for the wireless, no spill-through keyboard (wtf guys, that was brilliant!). About 6 months in, the video glitches started: there is a short which corrupts video memory while using wifi. The fix, of course, is a replacement system board, which is great if you have a spare third-gen X1 Thinkpad Carbon lying around and can just up and change workstations like it's no big. Workaround: rfkill block wifi, or just get used to the strobe effect.

The latest: Thinkpad X1 Yoga. And let's just put it out there: this doesn't work out of the box like the previous two did. Lenovo and Intel seem to have backed off of their Linux commitment, though that's probably half their users these days (errybody else on the mobile, right?)

Here's the rundown on getting the X1 Yoga up and running with a Linux install.


Basic Linux

Debian 8.3.0 AMD64 netinst ISO if you were wise enough to buy the OneLink+ RJ45 ethernet adapter; otherwise, use the regular (non-netinst) ISO.

Use unetbootin to make a USB flash drive image for this. Go into the BIOS on the X1 Yoga, enable F12 for boot device selection (just because), turn off secure boot, maybe enable diagnostic boot as well. No need to allow legacy BIOS installs, Debian will work just fine with uEFI (unlike FreeBSD where you need to use the 10.3-Beta or 11 memstick). Go through the install like you usually do. Your ethernet (if you have it) will be detected (again, unlike in FreeBSD).

Sound, APCI (suspend, hibernate, and battery status) and video card should all be working out of the box.


Wireless

Upgrade your kernel to 4.4.1 from the ubuntu PPA:
  http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.1-wily/
This is per notes about a typo in the iwlwifi driver at
 https://bbs.archlinux.org/viewtopic.php?id=206831
Looking at the kernel source, 4.2.7 has the fix as well, but might as well go with the latest stable release of the kernel if we're upgrading.

You'll also need to upgrade the firmware, or your Wifi will not be recognized. There is a debian package in backports for this:
 https://packages.debian.org/jessie-backports/firmware-iwlwifi


Skylake P-State Fix

If you rebooted excitedly into your new kernel without following the rest of these steps, you're likely looking at a blank screen. That's because of a skylake p-state bug reported here:
 https://www.reddit.com/r/thinkpad/comments/410haw/yoga_260_wont_boot_ubuntu_with_kernel_4244_but/

This is because the kernel needs a command line parameter to not totally crap itself when it encounters Skylake's poor implementation of pstate.

To fix this, change the cmdline in /etc/default/grub to:
  GRUB_CMDLINE_LINUX_DEFAULT='intel_pstate=no_hwp quiet'

Don't forget to run update-grub when you're done.

Hopefully this will go away in 4.5; according to a commenter, that's what the kernel guys are trying for.


i915 Xorg Video fix

Of course, the fancy new kernel hosed your X windows, didn't it?

Not sure why there's such a tight coupling there, but never mind. This one's an easy fix, per http://comments.gmane.org/gmane.linux.debian.user/508779 . Download (and install) a new version of the xserver-xorg-video-intel package from backports:

https://packages.debian.org/jessie-backports/xserver-xorg-video-intel


Touchscreen

The touchscreen is recognized and configurable in X Windows, but does not respond to touch or stylus events. A bug report is open at the linuxwacom project.

The touchscreen (+ stylus) works out of the box once the firmware update is installed (via windows, sadly,so don't pave the drive until you've updated firmware). Make sure to perform a hard reset (60 sec with a paperclip in the bottom of the laptop) after the upgrade.

To disable the touchpad when entering tablet mode (note: the keyboard will be disabled by  the BIOS), create the file /etc/acpi/events/thinkpad-tablet-touchpad-off with the following contents:

event=ibm/hotkey LEN0068:00 00000080 000060f0
action=/usr/local/bin/touchpad_disable.sh


You can verify the hotkey parameter by running acpi_listen and folding the display back to tablet position.

Next, create the file /usr/local/bin/touchpad_disable.sh with the following contents:

export XAUTHORITY=`ls -1 /home/*/.Xauthority | head -n 1`
export DISPLAY=":`ls -1 /tmp/.X11-unix/ | sed -e s/^X//g | head -n 1`"
tp_dev='SynPS/2 Synaptics TouchPad'
state=`xinput list-props "$tp_dev" | grep 'Device Enabled' | cut -d':' -f2 | tr -d '\t '`
if [ $state -eq 1 ]
then
  xinput disable "$tp_dev"
else
  xinput enable "$tp_dev"
fi

You will need to give root access to the X server session for this. The quickhack version is to add xhost + & to ~/.xsessionrc, but it's better to do something like +root@localhost..


The Acer BMA150 accelerometer is detected as a joystick device, so various /sys/bus/iio -based screen orientation scripts like thinkpad-yoga-scripts won't work. An alternative is to define a Window Manager screen edge event that invokes a script to cycle through screen orientation rotations:

#!/bin/sh
# script to rotate screen (counter-clockwise) based on current orientation

conn_line=`xrandr | grep ' connected'`
dev=`echo "$conn_line" | cut -d' ' -f 1`
orient=`echo "$conn_line" | cut -d' ' -f 5`

case $orient in
  '(normal' )
    xrandr --output $dev --rotate left
    ;;
  'left' )
    xrandr --output $dev --rotate inverted
    ;;
  'inverted' )
    xrandr --output $dev --rotate right
    ;;
  'right' )
    xrandr --output $dev --rotate normal
    ;;
  * )
    xrandr --output $dev --rotate normal
    ;;
esac

Some other approaches are discussed in this Manjaro Linux post.


Fingerprint reader

Is not supported by fprint. USB ID: 138a:0090 (Validity, no product string in usb-devices).


Camera

Works out of the box. Driver is uvcvideo. VLC mirrors it without any problems:

vlc v4l:// :v4l-dev="/dev/video0" :v4l-adev="/dev/audio"


FreeBSD

You need to use a 10.3 (BETA3 or more current) UEFI memstick image to do the install with, of course, the OneLink ethernet adapter connected. Presently, no support for wifi or Xorg support (perhaps in 11?). To boot, add an entry in /etc/grub.d/40_custom as follows:

menuentry "FreeBSD 10.3-BETA3" {
  insmod chain
  set root=(hd0,gpt10)
  chainloader (hd0,gpt10)/boot/loader.efi
}

...replacing gpt10 with the partition number where FreeBSD was installed. This starts counting from 1, so the example here is the tenth partition on the first hard drive (hd0).

Note that for update-grub to correctly recognize the FreeBSD partition (and therefore find loader.efi), you should provide an fstab entry to specify the UFS partition type:

/dev/sda10      /home/bsd       ufs    ro,noauto,ufstype=ufs2       0       0

Sunday, September 6, 2015

Using a webapp as a local application GUI

This is just a quick proof-of-concept for using a local webserver and a browser instance as a standalone GUI application. The idea is to use a browser and an embedded webserver as your UI toolkit, instead of hoping that the user was able to get qtbindings or shoes installed.


Http Application

The main application spawns two child processes: a local web server and a web browser instance.

In an ideal world, the application would wait for both processes to exit, and perhaps offer to restart one or the other in the event of a crash. Because of the way modern browsers work -- opening URLs in an existing process instead of starting a new one -- this has to be simplified to waiting only for the web server process to exit, and hoping that the browser process can successfully manage things.

The HttpApplication class provides a run() method that starts a webserver on localhost using the first available port, then opens a browser window/tab connected to the webserver:


class HttpApplication

  def run(host=nil, port=nil, browser=nil, path=nil)
    @server = ServerProcess.new(host, port, @controller)
    @server.start
    uri = server.uri.dup
    if path
      path = '/' + path if (! path.start_with? '/')
      uri.path = path
    end
    Browser.open(uri, browser)
    Process.waitpid(@server.pid)
  end
end

When the webserver process exits, the application will exit. Note that this means the webapp itself must close the webserver process, either through an explicit signout, or through a Javascript on_close or on_unload event handler.


ServerProcess

The webserver process is simply a Webrick instance with a servlet acting as a controller. This is pretty straightforward: the only interesting part is detecting the next available port so that it can be passed to the Webrick constructor.

require 'uri'
require 'webrick'
class ServerProcess
  attr_reader :host, :port, :controller, :pid, :uri, :webrick

  def initialize(host=nil, port=nil, controller=nil)
    @port = port || get_avail_port(host)
    @host = host || IPSocket.getaddress(Socket.gethostname)
    @controller = controller
  end
  def start
    @pid = Process.fork do
      @webrick = WEBrick::HTTPServer.new( :Port => @port )
      @webrick.mount('/', @controller) if @controller
      trap('HUP') { @webrick.stop; @webrick.start }
      trap('INT') { @webrick.shutdown }
      @webrick.start
    end

    trap('INT') { Process.kill 'INT', @pid }
    trap('TERM') { Process.kill 'INT', @pid }
    @uri = URI::HTTP.build( {:host => @host, :port => @port} )
    self
  end
  def stop
    @webrick.shutdown if @webrick
    Process.kill('INT', @pid) if @pid
  end


  private
  def get_avail_port(host)
    host ||= (Socket::gethostbyname('')||['localhost'])[0]
    inf = Socket::getaddrinfo(host, nil, Socket::AF_UNSPEC,
Socket::SOCK_STREAM, 0, Socket::AI_PASSIVE)
    fam = inf.inject({}) { |h, arr| h[arr[0]]= arr[2]; h }
    sock_host = fam['AF_INET'] || fam['AF_INET6']
    sock = sock_host ? TCPServer.open(sock_host, 0) : TCPServer.open(0)
    port = sock.addr[1]
    sock.close
    port
  end
end


Browser

The browser process  is a simple launcher that opens a URL in a browser instance , in what is hopefully a platform-independent manner. One could use the launchy gem or some other suitably over-engineered solution, but there's really not much need to support more than Windows, OS X, and Linux. Let's face it, anyone who doesn't have xdg-open installed probably doesn't want to run your webserver-based app on their ideologically-pure system.

require 'shellwords'
class Browser
  if RUBY_PLATFORM =~ /darwin/
    URL_OPEN_COMMAND = 'open'
  elsif RUBY_PLATFORM =~ /linux/ or RUBY_PLATFORM =~ /bsd/
    URL_OPEN_COMMAND = 'xdg-open'

  else
    # assume windows
    URL_OPEN_COMMAND = 'start'
  end
  def self.open(uri, cmd=nil)
    pid = Process.fork do
      uri_s = Shellwords.shellescape uri.to_s
      `#{Shellwords.shellescape(cmd || URL_OPEN_COMMAND)} #{uri_s}`
    end
  
    Process.detach(pid)
  end
end


RestContoller

The meat of the UI is in the controller object which HttpApplication uses to handle webserver routes. This particular one is based on Webrick's AbstractServlet class, and uses regex patterns to determine the handler for a particular route.

Each route consists of a pattern and a block. The pattern is used to match the URL; for best results, the ^ and $ anchors should be used, and the URL path should be absolute (i.e., starting with '/').

The block receives three arguments: query, request, and server. The query is a Hash extracted from the Webrick HTTPRequest object, the request is the complete HTTPRequest object, and the server is the Webrick HTTPServer object for the webserver (used mainly to shutdown the server). The block must return either the response body or an array [body, content_type, status]; these return values will be written to the Webrick HTTPResponse object.

class RestController < WEBrick::HTTPServlet::AbstractServlet
  DEFAULT_STATUS = 200
  DEFAULT_CONTENT_TYPE = 'text/html'
  DEFAULT_BODY = '404 : not found' # this should really be html

  @routes = []
  def self.route(pat, &block)
    @routes << [pat, Proc.new(&block)]
  end
  def self.routes; @routes; end
  def initialize(server)
    @webrick = server
    super server
  end
  def fill_response(response, bodycontent_type=DEFAULT_CONTENT_TYPEstatus=DEFAULT_STATUS)
    response.status = status
    response['Content-Type'] = content_type
    response.body = body
    response
  end
  def route_request(request)
    content_type = DEFAULT_CONTENT_TYPE
    status = 404
    body = DEFAULT_BODY
    self.class.routes.each do |pat, proc_obj|
      if request.path =~ pat
        arr = [ proc_obj.call(request.query, request, @webrick) ].flatten
        body = arr[0]
        content_type = arr[1] || DEFAULT_CONTENT_TYPE
        status = arr[2] || DEFAULT_STATUS
        break
      end
    end
    [body, content_type, status]
  end
  def do_GET(request, response)
    body, content_type, status = route_request(request)
    fill_response(response, body, content_type, status)
  end
  def do_POST(request, response)
    body, content_type, status = route_request(request)
    fill_response(response, body, content_type, status)
  end
end


Example

Here's a quick example that defines a few handlers.

def create_rest_controller
  cls = RestController

  # /test
  cls.route(/^\/test$/) { |q,r| ["test", "text/plain"] }
  #/quit
  cls.route(/^\/quit$/) { |q,r,s| s.shutdown; ["shutting down", "text/plain"] }

 

 #/
  cls.route(/^\/$/) { |q,r,s| s.shutdown; ["root!""text/plain"] }
  cls
end

app = HttpApplication.new( create_rest_controller )
app.run

The content returned by the routes is all plaintext because hey, the blogger interface sucks for entering < and  > characters.

Note the /quit handler: this is necessary to shutdown the webserver process (otherwise it will continue to run in the background). If you are certain your users permit Javascript, you can have an event handler hit this URL when the user closes the browser tab/window.

An alternative is to have the web server manage a PID file, so that subsequent invocations of the application will connect to the webserver process running in the background (much like the browsers that forced us into this mess).

As usual, the code is available on github.