jueves, 19 de enero de 2017

gnu parallel as a queuing system

This post is about a gnu parallel, a tool I recently discovered, and I'm starting to use a bit more every day.

At its core, it's a command to multiple commands in parallel, but it has many many different options to customize how the paralelization is done, notifications, and other configs. Take a look at the official tutorial or the man page, which contain a wealth of info and examples.

Let's get SICP videos

The use case I have today is to use it as a simple queuing system. I just want processes to start when I have a new job for them.  The task at hand is to download all sicp lectures, at one download at a time (don't want to hog the network).

First of all, we notice the pattern of the links:

  1. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec1a.mp4
  2. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec1b.mp4
  3. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec2a.mp4 
  4. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec2b.mp4 

We notice the pattern, right? Let's craft some generator for the filenames.

 perl -e 'for(1..15){for $i (('a','b')){print "wget http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec$_$i.mp4\n"}}' >sicp.list

After we generated the file, we're going to run the command in the following way:

cat sicp.lisp | parallel -j1 --tmux

this makes parallel run one after the other, and just putting the outputs of each job in a tmux tab.

B, b, but.... what's the point of all this?

Ok, we didn't use parallel for anything useful, we could have run the list as a shellscript, and be happy.   The idea is that we can use this simple mechanism to treat the file as a job queue, that waits for new incomming jobs and then processes them.  Beware, it has very little logging, and you can't do very sophisticated error recovery (see --resume-failed), so it's NOT a replacement for resque/sidekiq/etc...  In fact, I'd love to see something like a suckless queuing system based on parallel and bash.

So, here's the command to use this tool as a queuing system.

touch joblist; tail -f joblist | parallel --tmux 

Then, add lines to the joblist for them to be executed. It's a very easy plumbing task:

echo "sleep 10" >joblist

tail -f works as our event loop, waiting for new tasks to come, and it will pass the tasks to parallel, that will apply the job contention depending on the number of jobs you configure it to run in parallel.

I've just scratched the surface of what parallel is able to do. Do some searching around, and take a look at the man and tutorial to get a grasp of what this amazing gnu tool is able to do.  Taco Bell programming at its best!


Here I'm pasting some useful refs (appart from the ones already mentioned). 

martes, 20 de diciembre de 2016

How to replace procrastination by a list of parsing techniques

Here's a very nice Wadler's paper with quite simple ideas to get you through a way of parsing which I guess predates monads, and monadic parser combinators.

How to replace failure by a list of successes.

Seeing it after playing with SMUG (Common Lisp Monadic parser combinator lib), watching MJD's Higher Order Parsing techniques with perl, and re-reading his parsing chapter definitely helped a lot to make all that fuzzy knowledge settle a bit.

Unfortunatelly, I'm not using any parsing technique very often, but I like to read about all those crazy parsing techniques. (See META: Pragmatic parsing in Common Lisp for another one (inspired by Val Schorre's no-words-to-describe-how-enlightening-is-it META-II)).

Joining the monad party

Ok, so I won't try to explain what is a monad, or where is it used, mainly because I'm not sure I know the answers to these questions, but I'm just going to list some links that seemed to make sense (more or less) at some point. I think 'programmable semicolons' is a very nice way to put it.

- http://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf
- http://blog.sigfpe.com/2006/08/you-could-have-invented-monads-and.html
- https://codon.com/refactoring-ruby-with-monads
- http://stackoverflow.com/questions/28139259/why-do-we-need-monads
- https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern
- http://blog.reverberate.org/2015/08/monads-demystified.html
- https://www.schoolofhaskell.com/school/advanced-haskell/functors-applicative-functors-and-monads
- http://stackoverflow.com/questions/44965/what-is-a-monad

Monads in CL/scheme
- https://www.lrde.epita.fr/dload/papers/newton.16.monad.report.pdf
- http://www.kylheku.com/cgit/lisp-snippets/tree/monads.lisp
- https://common-lisp.net/project/cl-monad-macros/monad-macros.htm
- http://www.ccs.neu.edu/home/dherman/browse/shared/notes/monads/monads-for-schemers.txt

Monads in Clojure
- https://github.com/clojure/algo.monads

- http://fsharpforfunandprofit.com/rop/
- https://www.schoolofhaskell.com/school/advanced-haskell/functors-applicative-functors-and-monads
- http://www.valuedlessons.com/2008/01/monads-in-python-with-nice-syntax.html

Seem cool, but couldn't grasp it:
- http://www.eliza.ch/doc/wadler92essence_of_FP.pdf
- https://ncatlab.org/nlab/files/WadlerMonads.pdf

Meta, why there are sooo many monad tutorials:
- https://byorgey.wordpress.com/2009/01/12/abstraction-intuition-and-the-monad-tutorial-fallacy/

miércoles, 7 de diciembre de 2016

Alan Kay interviews and more ideas

Recently, Alan Kay (you know, my 1st reference for maaany things) appeared in Code Mesh and Joe Armstrong (of Erlang fame) interviewed him. As always, lots of insights and some old concepts mixed with new ones. 

https://www.youtube.com/watch?v=fhOHn9TClXY . HN discussion here : https://news.ycombinator.com/item?id=13033299

Also here is a 2h+ long interview from 1990 that I think appeared recently on the internet.

More 'Kayisms', some thoughts, reflections and examples on how to store data that can last (as in be read) practically forever.
http://www.vpri.org/pdf/tr2015004_cuneiform.pdf . (related to http://worrydream.com/TheWebOfAlexandria/ )

And here there are a couple of other links I hadn't seen about him, the story of personal computing, and progress/critical thinking  in general.

domingo, 27 de noviembre de 2016

Install Wifislax 4.12

Putting wifislax in a usb stick is a bit unlike most other distros.

  1.  copy it with dd, or unetbootin. 
  2. run ./boot/Linux_Wifislax_Boot_Installer.com
  3. when it fails with "error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory".
  4. ln -s /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 /usr/lib/libdevmapper.so.1.02
  5. rerun  ./boot/Linux_Wifislax_Boot_Installer.com 
  6. Audit!

martes, 22 de noviembre de 2016

TIL: Oneliner http endpoint

Today I was looking for some 'Taco Bell' way to create an http endpoint in a server that would output very basic stats about the server.

In this stackoverflow thread I found a pretty neat way to do it.

 while true ; do nc -l -p 1500 -c 'echo -e "HTTP/1.1 200 OK\n\n $(lsof -a -p $(pgrep myprocname) | wc -l)"'; done

Nice, eh?

miércoles, 19 de octubre de 2016

TIL: highlight jira's comments

At (sys:getenv "WORK") we use JIRA as an issue tracker.  An issue tracker you can hate as much as any other issue tracker (maybe even a bit more).

The problem at hand today is that I can't see the highlighting of the focused element in jira in my screen (depending on the lights and reflection), and in long threads, it makes impossible to see what is the comment that I'm supposed to be looking at.

The tip for today is to overwrite css for some sites.  In conkeror we could add the configs in .conkerorrc but it seems firefox (keysnail for that matter) makes us use a userContent.css which belongs to a profile.

For me, I had to edit the file "/home/rgrau/.mozilla/firefox/c32dsif..default/chrome/userContent.css"

and add this:
@-moz-document domain(jira.myserver.com) {
    .focused { border: solid 1px !important; }
    /* .focused { background-color: red !important; } */