Chris McCormick - News http://mccormick.cx/news/ Chris McCormick - News en Copyright 2008- Chris McCormick 60 GMT chris@mccormick.cx mccormick.cx/news/ A Clojure-like Lisp That Runs In Bash entries/a-clojure-like-lisp-that-runs-in-bash http://mccormick.cx/news/entries/a-clojure-like-lisp-that-runs-in-bash wordmark.svg screencast.svg

Fleck is a Clojure-like LISP that runs wherever Bash is. Get it here.

This is a little experiment I hacked together from the amazing make-a-lisp project. My hard drive is littered with attempts to make this exact thing several times, and this was the first time I got it working properly.

My friend Crispin reminded me of this idea when we were discussing lightweight Clojure based scripting tools for domain specific tasks. He has made a fantastic Clojure static site generator called bootleg on top of GraalVM. We've both been inspired by the work of Michiel Borkent such as Small Clojure Intepereter and Babashka.

For more Clojure-likes check out awesome-clojure-likes.

Exciting times in the Clojure-verse!

]]>
/tags/all Sat, 30 Nov 2019 07:25 GMT
A Week in Singapore entries/a-week-in-singapore http://mccormick.cx/news/entries/a-week-in-singapore singapore.jpg cyber.jpg asimov.jpg veg1.jpg map.jpg watch.jpg

Last week I was in Singapore with my friends PVI Collective, consulting with local artists Ekamatra and Drama Box. We were hacking on eachother's artworks in their studio loft above Chinatown. Locative code, smart watches and dark LARP-arts - it was like I'd stepped into a post-cyberpunk William Gibson novel somehow.

I also gave a talk at the Singapore JS meetup on Bugout, the library I wrote to help build decentralized web applications. You can watch the talk here:

https://www.youtube.com/watch?v=zfPX-dR6LeM

This trip gave me the headspace to think about priorities and progress on my side projects. At the moment I'm focused on two main things apart from paid work.

Firstly, I'm working towards the launch of the paid version of SVG Flipbook, an SVG animation utility for people using Inkscape and Illustrator. Hopefully this will be ready for launch in the next couple of weeks. This is a project I started a while ago and I'm coming back to now.

Secondly, I had some time to think about and work on WebRTC Signaling Mesh, which is coming along nicely. It will be a way to do WebRTC signaling without resorting to centralized services. I've had the design floating around my head for a long time and I've finally begun implementation. Will hopefully have an update with progress on that soon.

Thanks for tuning in!

]]>
/tags/all Thu, 21 Nov 2019 01:35 GMT
Announcing The Bugout Box entries/announcing-the-bugout-box http://mccormick.cx/news/entries/announcing-the-bugout-box A few weeks ago at the BSides Perth conference I announced this piece of hardware I've been tinkering on.

The Bugout Box is a decentralized web appliance. Its a Raspberry Pi that any browser can connect to from anywhere in the world over WebRTC - a censorship-resistant way to serve data, APIs and apps.

Examples of things you can do with the box:

Bugout Box 3d view

Head to bugout.network/box if you want to find out more and sign up to the pre-release list.

]]>
/tags/all Tue, 05 Nov 2019 02:29 GMT
How I built an Excel add-in to export HTML tables entries/how-i-built-an-excel-add-in-to-export-html-tables http://mccormick.cx/news/entries/how-i-built-an-excel-add-in-to-export-html-tables An economist told me the worst part of her job is turning Excel data into HTML tables, so I built an add-in to fix it.

Many software developers probably don't realise that Microsoft Office add-ins these days are simply web pages which run inside of a panel in the UI. I suppose it was to be expected given the trend of the last couple of decades towards web based everything, but it still came as a surprise to me when I had to build one for a job. In my mind Office was still back in the world of COM objects and Delphi and DLLs.

After discovering how easy it is, I decided to do it again for this side project. Here is how I did it.

The plugin (whoops, "add-in") I built is called "Excel to HTML table" and does what it says on the tin. You make a selection of cells, click "copy", and you get a clipboard full of the corresponding HTML code that will render those cells. After that you can paste the code into your text editor and use it in a site's HTML.

Excel to HTML table screencast

Microsoft have some nodejs and yeoman projects to help you get up and running but I'm the sort of developer who likes to roll their own and keep things tight, tidy, and tiny. I like to build things from first principles so that I can understand what is going on at as low a level as possible. Here's what I discovered.

The Microsoft Tutorial has a lot of good info in it, but it's geared towards people using either Node + Yeoman or Visual Studio. If you're a text-editor-and-command-line person like me the best way to get started is just to grab the example .html, .css, and .js snippets from the Visual Studio version.

Get a dev server running

Any local HTTPS server will do. The difficult part is it must be HTTPS, even on localhost, or Word/Excel will refuse to load your add-in. My add-in is almost entirely client-side code and so I could get away with a small Python server using the built-in libraries like this:

from http.server import HTTPServer,SimpleHTTPRequestHandler httpd = HTTPServer(address, SimpleHTTPRequestHandler) httpd.socket = ssl.wrap_socket(httpd.socket, keyfile='selfsigned.key', certfile='selfsigned.cer', server_side=True) httpd.serve_forever()

As you can see this looks for the .key and .cer files in the local dir, and you can create those with openssl:

openssl req -x509 -newkey rsa:4096 -keyout selfsigned.key -out selfsigned.cer -days 365 -nodes -subj "/C=US/ST=NY/O=localhost/OU=Localhost"

You can do the same thing in nodejs with express if that's your bag by passing the right options to https.createServer:

const server = https.createServer({ key: fs.readFileSync('./selfsigned.key'), cert: fs.readFileSync('./selfsigned.pem') }, app).listen(8000)

Before you can load your add-in you should make sure your browser has accepted the self-signed cert or the add-in won't load. Do this by browsing to your localhost:8000 server and bypassing the certificate warnings.

If you're debugging in the native Office app instead of Office Online, you will need to do this in Internet Explorer for it to work as far as I can tell.

Manifest validation

add-in.gif

Office finds out about the URL for your add-in using a "manifest" file. This file is XML and pretty fragile. You need to make sure it complies with the spec. Luckily Microsoft have a tool for verifying the add-in on the command line. Install the npm package "office-toolbox": "0.2.1" and then you can run a command like this:

./node_modules/.bin/office-toolbox validate -m ./path/to/manifest.xml

This will report most issues that come up.

In the manifest you can use https://localhost:8000/Home.html and it will point to your local dev server when running.

The Code

The code itself is basically web code. You can use libraries like React and jQuery in your UI. The exception of course is when calling the native APIs. These are exposed through an interface like Excel.run(...) and make heavy use of promises for async. You will often find yourself doing context.load() and then waiting for the promise to resolve before doing the next thing in the document. The API documentation is super useful for figuring out what is possible and how to do it.

Debugging

When it comes to iterating on your code and debugging, by far the easiest way is to use Office Online. This is because it is already in the web browser so debugging works the same way as you are used to - the add-in is just an iframe.

I was even able to do my dev & debugging right at home in Firefox on GNU/Linux!

At some point you will want to debug using an actual copy of Office running on Windows. I used Office 2016 on my wife's computer.

If you are not a Windows developer the following tip will save you a lot of time when it comes to debugging native. It's called "F12 Developer Tools" and it's buried deep inside a Windows subdirectory in C:\Windows\System32\F12\.

What this tool does is attach a "web console" type of debugger to your Office add-in instance running inside Office. You can do stuff like console.log and also see JavaScript errors which are thrown.

Ready to roll

8e9a6781754c869c84202121fb7a2fef.png

Once you've got those pieces in place you should have a basic add-in up and running. After that it's all about referencing the docs to figure out how to do the thing you want your add-in to do.

Submit

Finally when you're ready to ship, you submit your manifest.xml to the App Source "seller dashboard". Expect to wait a few days for the validation team to get back to you, and to have to fix things. This process was useful as they see the software with a fresh pair of eyes and give actionable feedback.

LISP madness

So finally I should mention my LISP obsession. I actually wrote all the code for this plugin in a LISP called Wisp. It's a Clojure-like that compiles to JavaScript and seems quite similar to ClojureScript but with three core differences:

  1. It lacks almost all of the great features & tooling of ClojureScript.
  2. It is closer to being "JavaScript with LISP syntax".
  3. It compiles down very small if you know what you are doing. How small? My final compiled .js bundle for this add-in is just 8.2k non-gzipped.

So that's about it. I hope you got something out of this article on building Microsoft Office plugins using web tech.

Now back to open source dev for me. :)

]]>
/tags/all Mon, 30 Sep 2019 12:25 GMT
How To Make Hy-lang More User-Friendly entries/how-to-make-hy-lang-more-user-friendly http://mccormick.cx/news/entries/how-to-make-hy-lang-more-user-friendly Hy(lang) is a LISP-family programming language built on top of Python. You get the rich Python language & library ecosystem, with a LISP syntax and many of the language conveniences of Clojure, such as reader macros for easy access to built in data types. What's not to love?

97bb1a75b5a52e8c6a0a8e7e68cd4b4f.png

I recently found out I have the most GitHub stars for projects written in Hy of any developer worldwide. With this admittedly ridiculous credential in hand I'd like to offer some opinions on the language.

I really like Hy a lot. I prefer writing code in Hy to writing code in Python, and I am writing this post because I want to see Hy do well. I originally wrote this as a list of things in Hy that could possibly be improved, from the perspective of an end user such as myself. Then my friend Crispin sent me this fantastic video by Adam Harvey: "What PHP learned from Python" and I realised that all of the issues I have with Hy stem from the same basic problem that Adam talks about.

Here is the crux of the problem: I've written a bunch of software in Hy over the past few years and often when I moved to a newer point-release of the language all my old code breaks. My hard drive is littered with projects which only run under a specific sub-version of Hy.

gilch[m] It might depend on the Hy version you're using.

I understand that the maintainers of the language, who are hard-working people doing a public service for which I'm deeply grateful, are concerned with making the language pure and clean and good. I understand that languages have to change to get better and they need to "move fast and break things" sometimes. That's all fine and good.

However, I think Adam Harvey's point stands. If you want users to use and love your language:

  • Break things cautiously
  • Maintain terrible things if it makes life better
  • Expand the zone of overlap [backwards compatibility between consecutive versions]

I think if you can maintain backwards compatability you should.

Almost all of the breakages I've experienced between Hy versions could have been avoided with aliasing and documentation. Not doing this backwards compatibility work basically tells your users "don't build things with this language, we don't care about you." I know that is almost certainly not the attitude of the maintainers (who are lovely, helpful people in my experience) but it is the way it comes across as an end user.

Here is a list of things which have changed between versions which blew up my Hy codebases each time I upgraded Hy:

  • Renaming true, false, and nil to the Python natives True, False, and None. These could have been aliased to maintain backwards compabitility.

  • Removing list-comp and dict-comp in favour of the excellent lfor, dfor, and friends. Again, small macros could have aliased the old versions to the new versions.

  • Removing def in favour of setv. A very small macro or maybe even an alias could have retained def as it is pretty much functionally identical from the perspective of an end user.

  • Removing apply, presumably in favour of #**. Support could have been retained for the functional apply. There are situations where a proper apply is favourable.

  • Removing the core macro let. It seems there was an issue where let would not behave correctly with generators and Python's yield statement. A more user-friendly solution than chopping the imperfect let from the default namespace and breaking everybody's code would have been to document the issue clearly for users and leave the imperfect let in. I know it is available in contrib. Moving it to contrib broke codebases.

Having your old code break each time you use a new version is frustrating. It makes it hard to justify using the language for new projects because the maintainance burden will be hy-er (sorry heh).

Some other minor nits I should mention which I think would vastly improve the language:

  • loop/recur should be core. Like let these are available in contrib, but that means you have to explicitly import them. Additionally it would be super nice if they were updated to support the vector-of-pairs declaration style of let and cond etc.

  • Why does assoc return None? This is completely unexpected. If there are performance issues then create an alternative which does what Python dict assignment does (aset?) but it seems unwise to break user expectations so fundamentally.

36583050842532df644f183cc62890e7.png

It is much easier to provide criticism than it is to write working code. I am sorry this is a blog post instead of a pull request, and I hope this criticism is seen as constructive. I want to thank everybody who has worked on hy. I am a huge fan of the language. It is an amazing piece of software and I am very grateful for your work.

]]>
/tags/all Thu, 26 Sep 2019 13:20 GMT
Speaking schedule 2019 and beyond entries/speaking-schedule-2019-and-beyond http://mccormick.cx/news/entries/speaking-schedule-2019-and-beyond I've got three conference talks coming up in Perth (Australia), London, and The Gold Coast (Australia). If you're nearby let me know - I would love to buy you a coffee/beer and hear what you're up to.

Security BSides

This Sunday, September 22nd, Perth, Western Australia.

I'm presenting "Bugout: practical decentralization on the modern web." It's a talk about the library I built on top of WebTorrent for building web based decentralized systems.

Bsides Perth Logo

Clojure eXchange

December 2nd-3rd, London, UK.

I'm giving a keynote: a show and tell of the multitude of strange things I've been building with the Clojure[Script] family of programming languages, and how Clojure enables the bad habit of starting way too many projects. I'll also give an update on Thumbelina, the tiny MIDI controller I've been working on with my friend Dimity.

Skills Matter Clojure eXchange

Linux.conf.au

January 13th-17th, Gold Coast, Australia.

I'm talking about Piku, and how it helps you do git push deployments to your own servers. I've made a bunch of contributions to this open source project in recent months. I've personally found a huge productivity gain from being able to deploy internet services without having to think too much, and I'm excited to show others this too.

Linux Australia Logo

]]>
/tags/all Wed, 18 Sep 2019 07:50 GMT
Droneship Study entries/droneship-study http://mccormick.cx/news/entries/droneship-study a9bbb77a696a84276d8e05ed6a6867b1.png

Droneship. Study in the style of thisnorthernboy.

]]>
/tags/all Sun, 08 Sep 2019 06:54 GMT
Notes on "History of the Blockchain" by Nick Szabo entries/notes-on-history-of-the-blockchain-by-nick-szabo http://mccormick.cx/news/entries/notes-on-history-of-the-blockchain-by-nick-szabo In November 2015 Nick Szabo gave a talk on the history of the blockchain which was dense with useful ideas.

61120c710d06cf0ec63488c1abd14665.png

Here are some notes I took on his talk:

  • Philosophical inspiration to Cypherpunks who invented Cryptocurrency:

    • Ayn Rand: Galt's Gulch - independence from corrupt institutions.
    • Tim May: "protect yourself with cryptography" (cyber Galt's Gulch.)
    • Friederich Hayek: Institutions of property, contracts, money are actually important to human freedom.
  • Use computer science to minimize vulnerability to strangers.

  • Non-violently enforce the good services of institutions.

  • "Try to secure as much as possible" not just communication.

  • Cryptography: only secures communications from 3rd parties.

  • David Chaum: let's apply this to money too.

  • Centralization problem remained in digital cash startups.

  • Bad assumptions in computer security: trusted third parties like certificate authorities are secure.

  • Trusted third parties are security holes.

  • Centralization is insecure.

  • E.g. Communists were able to get stranglehold with just control of railroads, newspapers, radio.

  • Gold is insecure

    • Spanish looted Aztec gold, pirates looted Spanish gold.
    • Part of end of gold standard was German U-boat threat to British gold transportation.
    • Franklin Roosevelt's government confiscated gold.
    • In modern times xray machines detect gold easily.
  • Decentralization per computer science is much more automated & secure than traditional security.

  • CS decentralization can only replace small fraction of traditional security but with very high cost savings.

  • Traditional security isn't the protocol itself, requires strong external law enforcement.

  • Computer security can be secure across national borders instead of siloed inside jurisdictions.

  • Cryptocurrency helps solve this through decentralization.

  • Separation of duties: several independent people to perform a task to get it done.

  • Each node as independent as possible.

  • E.g. crude measure of independence: geographic diversity of nodes.

  • Number of nodes is only a proxy measure of decentralization.

  • Smart contract:

    • Long lived process or "distributed app".
    • Acts like a contract.
    • Performance, verification etc.
    • Generally 2 parties + blockchain (replacing TPP).
  • Wet code = traditional law. Dry code = smart contract.

  • Law is subjective, enforced with coercion, flexible, highly evolved.

  • Smart contracts are mathematically rigorous, cryptographically enforced, rigid, very new.

  • Law is jurisdicionally siloed, and expensive to execute.

  • Smart contracts are super-national & independent and low cost.

  • Seals in clay/wax were important when writing was invented: signature + tamper evident.

  • Modern seals at e.g. crime scenes: sealing door, evidence bag with numeric identifier.

  • Blockchain can keep secure log with both semantics (serial number) and proof of evidence (photo hash).

  • Put proof of evidence on blockchain as well as semantic reference for contract code to interface with.

  • Can secure physical spaces with same mechanism.

  • Proplets: blockchain can tell them which keys have which capabilities.

    • For almost any valuable property that can be controlled digitally
    • Example: Auto-repo collateral upon contract breach.
    • Example: creditors without access to offshore oil rig used as collateral.
  • Recent project:

    • Trust minimized token: secure property titles, colored coins. Securing transfer of ownership.
    • Trust minimized cash flows (dividends, coupons, etc).
  • Idea: social networks for blockchains. Execute payment swaps & smart contracts after linking social accounts together.

  • Let's try to think about security more broadly instead of only encryption.

  • Let's try to protect everything that's important to us, without centralization.

]]>
/tags/all Wed, 28 Aug 2019 14:09 GMT
ClojureScript Pixel Game Engine With Blender Live-reloading entries/clojurescript-pixel-game-engine-with-blender-live-reloading http://mccormick.cx/news/entries/clojurescript-pixel-game-engine-with-blender-live-reloading Recently I've been hacking on a game engine for infinitelives called px3d.

around.gif

It's built on top of ClojureScript, Blender, and Three.js and it runs in the browser.

One feature I'm particularly happy with is the live-reloading of Blender assets into the game. You hit "save" in Blender and the updates appear in the running game a second later - no need to re-compile or re-load the game.

live-reloading.gif

The way this works is with a background script which watches the assets.blend file. It re-builds the assets.glb whenever it is modified, and writes the hash of the file into assets.cljs. Figwheel pushes changes to the compiled cljs files whenever they change, and there is another bit of code which tells three.js to re-load assets.glb if the hash has changed.

Infinitelives

Infinitelives is the vehicle me and my buddy Crispin use to make games and tooling, mostly for gamejams. The gamejam format is great because it is time-boxed, which means we can periodically do this self-indulgent thing we enjoy without taking too much family or work time.

Gamejams are typically only 48 hours long and so we have learned some good techniques for shipping working code under extreme constraints. A hardcore economy of time, resources, and scope is required.

ClojureScript & Figwheel are perfect for this with their hot-loading of modified code. I built the tight Blender re-load integration for the same reason. Hand drawn graphics consume a lot of time during jams and this should help us really level up on the content side of things.

cda02a53bdeea5ee13ac8d943761a42b.png

If you'd like to find out when we release games and new tools you can sign up to our release notifications on the infinitelives home page or follow us on Twitter.

]]>
/tags/all Sat, 17 Aug 2019 05:35 GMT
Goomalling entries/goomalling http://mccormick.cx/news/entries/goomalling Image1802377562.jpg Image530038439.jpg Image1309211065.jpg Image-95081980.jpg Image-206257949.jpg Image-1571672220.jpg Image1085423041.jpg Image531224935.jpg Image-651670289.jpg Image-1045430088.jpg

]]>
/tags/all Sun, 04 Aug 2019 09:06 GMT
Joplin With Self-hosted Sync entries/joplin-with-self-hosted-sync http://mccormick.cx/news/entries/joplin-with-self-hosted-sync For some time I have been looking for a writing solution with the following properties:

  • Lets me review and make minor edits on my phone.
  • Is synched to my laptop where I can write longer form.
  • Supports simple markup such as Markdown.
  • Supports attachments and images.
  • Is Free & Open Source Software and can be self-hosted.

The solution is Joplin.

screenshot.png

It's a wonderful piece of software. There are apps for all of the usual platforms, including a direct link to the Android apk, which is a blessing if you are somebody who opts out of using Google services.

553e4b7b90a90a9132db5b59f75c84b3.png

Some other things which are great about Joplin:

  • You can edit notes in an external editor.
  • You can paste images directly into your document.
  • It imports and exports many formats including Markdown.
  • It can export individual articles to PDF.
  • Its native export format "JEX" is a simple tar file.
  • Its native data store is on-disk.
  • Sync is optional and very easy to set up.
  • Sync uses the widely supported webdav protocol.

Joplin Sync

I got sync between my devices working quickly by using Piku to deploy a simple webdav server to my Piku VPS. If you want to do this yourself, check out the latter repository and then push it to Piku as follows:

git remote add piku piku@MYSERVER.NET:webdav
git push piku master
piku config:set NGINX_SERVER_NAME=WEBDAV.SOMEDOMAIN.NET PASSWORDS="username:password username2:password2 ..." FOLDERS="/joplin:/home/piku/joplin"

After that is up and running you can configure Joplin sync by selecting "webdav" on each device and then enter the URL WEBDAV.SOMEDOMAIN.NET/joplin/ and the username:password pair you specified above.

I wrote this post with Joplin.

]]>
/tags/all Thu, 01 Aug 2019 03:19 GMT
Hacksilver: technical details entries/hacksilver-technical-details http://mccormick.cx/news/entries/hacksilver-technical-details setup-thumbnail.jpg

Yesterday I released Hacksilver, an album of procedurally generated "algorave" music. Some people had questions about the technology used to write it so I thought I'd write this up.

The beats and melodies were generated using drillbit, a LISP codebase written in a Python variant called Hy. The project outputs Impulse Tracker mod files which are then played and mixed live.

The interesting parts of that codebase are in the generators folder. For example the drill-n-bass choppage generator is here.

Each generator has three functions:

  • make-sample-set: which generates IT wav tables that are used by the generator (e.g. individual drum kit or synth sounds)
  • make-pattern-settings: which sets up parameters & context that will be re-used by the pattern generator to provide similarity across pattern variations
  • make-pattern: which outputs the pattern data in a format easily consumed by the Impulse Tracker file writer

Mixing and live-effects are performed in Pure Data. Originally I was using a fully software based mixer. However I discovered that a nicer mode of operation is to have individual bits of sound generating/filter hardware chained together. So I started using this Raspberry Pi based mixer + FX unit from another project to mix live.

One other bit of software in there is jsfxr which is wrapped by the LISP code and outputs 8-bit synth sounds (which are then used by the pattern generator). Because the synth definitions are simple JSON hash maps there is a fun pseudo-evolutionary technique I was able to use where you interpolate between the values of two synth definitions to generate new sounds based on two synth definitions that you like.

Hardhat tracker module 
player

I also built a little hardware Impulse Tracker renderer based on a Raspberry Pi running XMP with my friend Dimity. It has a Pocket Operator style sync output and runs directly into the mixer that both share the same timing and the fx can be quantised to the music which is playing.

If you're interested in the music hardware that Dimity and I are building and selling you can stay updated at bzzt.studio.

In the image at the top of this post the hardware Impulse Tracker renderer is the little box on the right hand side. The RPi mixer/fx unit is to the top right of the C64 keyboard. The Korg Nanokontrol2 strapped to the C64 keyboard is controlling the fx and mixing parameters on the RPi. They keyboard itself was for playing live synth sounds (a very simple arpeggiating subtractive synthesizer built in Pd).

]]>
/tags/all Fri, 07 Jun 2019 02:24 GMT
Hacksilver: new algorave album entries/hacksilver-new-algorave-album http://mccormick.cx/news/entries/hacksilver-new-algorave-album HackSilver 💀 ⚔ by chr15m

I just released Hacksilver, a new album of procedurally generated music.

It uses a whole slew of weird tech to generate the beats, melodies, synth sounds including beat-generating LISP, 8-bit synth generating Javascript, and Pure Data for the mixing and mastering. One thing that was particularly fun was procedurally generating Impulse Tracker files.

Would appreciate a re-share if you know of anybody who might be into this type of thing.

Enjoy!

]]>
/tags/all Thu, 06 Jun 2019 06:58 GMT
BEP44 For Decentralized Applications entries/bep44-for-decentralized-applications http://mccormick.cx/news/entries/bep44-for-decentralized-applications Depiction of decentralized network

In 2014 Arvid Norberg and Steven Siloti came up with a BitTorrent extension called BEP44. The basic purpose of BEP44 is to allow people to store small pieces of information in a part of the BitTorrent network called the DHT. The DHT ("distributed hash table") is a key/value lookup table that is highly decentralized. Prior to BEP44 it was used to look up the IP addresses of peers from the hashes of the torrent they were sharing.

BEP44 introduces two new ways of storing key-value data in the DHT. The first is the ability to look up small bits of information keyed directly on their hash. The second allows for the storage of cryptographically authenticated data, keyed on the public key. What this means is if you have some public key K then you can look up authenticated blobs of data stored in the DHT by the owner of that key. This opens up a variety of useful abilities to people building decentralized applications.

For instance when you distribute a file on BitTorrent it is immutable - you can't change it - but BEP44 provides a way to tell people "hey, there is a new version of this file that I shared before which you can find over here" in a secure and authenticated way. It turns out that this basic mechanism can be used to build a wide variety of decentralized, authenticated functionality and people have built cool experiements like a decentralized microblog using it.

The purpose of this post is to show you how the datastructure works and how to apply it more widely in your own software.

If you just want a working implementation you can use, check out the decentral-utils JavaScript library which has a single-function implementation of the datastructure and algorithm which you can use in the browser. Web browsers unfortunately do not have direct access to the BitTorrent DHT, but the library is still useful for authenticating up-to-date data blobs that are passed around between browsers, for example over WebRTC.

What's good about BEP44

Most of the advantage of BEP44 comes from the cryptographic signing. What this offers is a way for somebody (let's call her Alice) to verify that a piece of data from somebody else (let's call him Bob) is authentic and that it has not been tampered with. You can do cryptographic signing without BEP44 of course.

However, the BEP44 specification brings some other features for building decentralized systems:

  • Namespacing against a public key.
  • Replay attack prevention.
  • A compare-and-swap primitive.

The namespacing feature means that a single public key can have a bunch of different data blobs that others can authenticate. Because the datastructure is keyed on both the public key and a "salt" field, a single public key can store multiple blobs with different "salt" values as a sub-key.

Replay attack prevention is accomplished with the sequence field, which in BEP44 is a monotonically increasing integer. The way a replay attack works is some adversary keeps an old copy of something you have signed and then when you are offline they send it again and pretend its a new message. Because the message is signed with your key people are fooled by the adversary into thinking the old data is current. In BEP44 the sequence field prevents this because the adversary will have a data blob with a lower sequence number than the last one you shared and they can't generate a new blob with a higher sequence number because they do not have your private key. So the replayed data blob will be rejected.

Finally, the compare-and-swap primitive works by specifying a previous sequence number that the current data blob should replace. Peers will reject data blobs which don't replace the current sequence number they hold. In this way it's possible to acheive basic synchronisation and atomicity of data blobs. Using compare-and-swap guarantees that the new value you want to insert into the DHT can be calculated based on up-to-date information.

Example usage

Imagine you take upon yourself the small task of building a decentralized social network. Users have a timeline of posts which they have written. When they write a new post it should be appended to their timeline and people should see the updated timeline with the new post.

In the centralized social network case this is easy. The social network provider TwitFace has an internal copy of the poster's timeline to which they add the new post. Followers are notified of the update and the updated timeline is sent to them by the provider. The way the authentication works in this case is the original poster has logged in with their password and so the provider knows who they are, but the receivers of the update must trust the provider.

Depiction of centralized post authentication

How do readers know that the post is authentic and comes from the original poster? The reader must trust the provider completely. They must trust that the provider is showing them the authentic timeline of messages, that the messages have not been modified, that fake messages have not been injected into the timeline, that new messages have been added to the feed in a timely fashion, and that no message has been censored and removed from the feed by the provider.

The problem with this model is that trusted third parties are security holes. Centralized social networks do modify things people have said. They do inject posts into people's timelines (ads and worse). They do change the order and timing of posts to suit their own goals. Perhaps worst of all, they do censor posts completely.

I won't get into the politics of censorship. Suffice it to say that censorship is fine and good right up until the point where your values differ from the entity doing the censoring. Values change, mysterious outside influences are myriad, and few people have complete alignment of values even with our noble corporate overlords in Silicon Valley.

The basic goal here is that if you've chosen to read somebody's timeline you can trust that the feed of posts you are reading from them is authentic and complete and as they intended.

Happily, we can use the BEP44 technique to route around the trusted-third-party centralized-social-network-provider security hole. BEP44 provides a way to receive an update and verify it cryptographically. It provides a way to know that this is the latest and complete version. It allows the verifier to do this using only the received data structure without requiring any kind of secret server side logic or password based authentiction like you would find in a centralized social network.

Depiction of decentralized post authentication

How it works

At the heart of the algorithm is a small datastructure which can be shared, updated by the sharer, and then shared again. Receivers can verify the updates are authentic. The idea is that you can share a small piece of authenticated data which points to a larger piece of immutable data. For instance, you can share an authenticated datastructure with the hash of a torrent containing all of your posts (an RSS feed for instance). Receivers then know that torrent is the current representation of your timeline of posts.

The datastructure has the following fields:

  • value
  • seq
  • cas [optiona]
  • salt [optional]
  • pubkey
  • signature

When a verifier receives a copy of this data structure they perform the following checks:

  • Is the size of the value field below the maximum size?
  • Does the value conform to the expected format / encoding?
  • Is the seq number higher than the last seq number I saw (if any)?
  • If cas is present does it match the previous seq number I have?
  • Is the attached signature made with the private key corresponding to the attached pubkey over the datastructure's value, seq, and salt fields?

In this final point they are checking that the structure has been digitally signed with the author's private key. This is accomplished by concatenating salt + seq + value and then checking that the signature is valid for that data and the given public key.

In this way verifiers can know that an update from a known keypair/identity is authentic and current.

Read more posts on the subject of cryptography & decentralized systems.

]]>
/tags/all Sun, 12 May 2019 03:28 GMT
Sally Kathryn (Laing) McCormick entries/sally-kathryn-laing-mccormick http://mccormick.cx/news/entries/sally-kathryn-laing-mccormick SALLY.jpg

My mother, Sally Kathryn (Laing) McCormick, passed away on the 25th of April, 2019. This is the eulogy I gave at the celebration of her life on Monday.

"Your mum is the nicest person I've ever met."

I've heard these words from friends and strangers countless times during my life. They are how I slowly came to understand that Mum was somebody extraordinary and not to be taken for granted. I'm so glad to have been able to express my own gratitude to her in recent years.

In Henry the Fourth Shakespeare has a turn of phrase: "like bright metal on sullen ground." Mum was bright metal on sullen ground. She was solid gold.

Everyone here will have their own fond, and if you knew her well, sometimes frustrating memories of my mum. I want to focus today on those things I came to really admire in her.

We all knew that warm side of her because she gave it freely, to everyone, everywhere she went. No matter who you were you would get a sweet smile, warmth, and praise. Perhaps fewer people here will know of her strength, energy, and willpower, her humour, her infinite gratitude, and her fundamentally uncomplaining nature.

In recent years I've found inspiration in the philosophy of the stoics and it is hard not to notice strong parallels with the way mum carried herself, always ready to help, always grateful, always giving. Standing straight, or held straight, by sheer will. Until around 11 each night when the universe would gently say "ok, it's time to sleep now Sally", and she would nod off where she stood.

And then, as my father knows all too well, was woken every morning by her two alarms, one at 4:30, so that she had "time to think", and one at 5 so that she could actually wake up.

But Mum was not a stoic, she was a Christian. Even though she was tested by the Good Lord with three opinionated, and two atheist sons, her faith never waivered in the slightest, even at the very end. As with everything about her, it held fast and true and uncompromising throughout her life. I am not a Christian but I admire her persistence greatly.

I am going to try to be more like my mother.

I'm going to try to be as grateful as she was. To remember that every morning I wake, every cup of coffee, every lego spaceship built with my kids, is a gift of limited supply. To delight and give thanks as she did in the smallest of things, common or uncommon. Any time I said thank you to mum, she would brush it off. Clearly, she thought it her basic duty to this world and the people in it to give everything she had with gratitude.

The most valuable thing each of us has to give is our time. Mum always had time for people and most especially for us her sons. In this too I hope to emulate her and give more time to my own kids, and to you, my friends and family, and the other people in my life, just as she did.

She never ever complained. Never. Instead she did whatever was in her control to help other people, and sometimes even things which were well outside her control. She knew that complaining accomplishes nothing but to make you feel worse and to burden those around. Mum never burdened anybody. If I need help I will ask for it. If I can fix something I will simply fix it, just as she did.

Mum had incredible energy. It wasn't the kind of energy you get from an energy drink, or from eating high energy food, because we all know she did not do that. No, it was the energy that springs forth at 4am at the hospital after several sleepless nights when your sick kid needs to be held. Its the energy you didn't realise you had until you decided to try and overcome the fatigue with willpower. Mum made amazing use of that particular reserve, and whilst it would probably not be advisable to dip in to quite the same extent as she did, it's good to know that there is always more you can give when required.

Finally, I aspire to mum's strength. She powered joyfully 100% into everything that life handed to her, and even her last hugs were vice-like, and with a genuine smile. Again that is not because she was particularly physically strong, although she was, but because she was emotionally and spiritually strong. If a mouse like my mother can be so mighty, surely I can too. She made it apparent that it is a simple matter of choosing to be strong.

Mum and Dad's legacy speaks for itself. The life they made for themselves in this country. The lives they helped others to build. The success of their sons Dirk and Mike in building a good life for themselves, sometimes against extraordinary odds which most people will never face. Of course I also owe a huge debt to my parents for the wonderful life I enjoy today, a debt I intend to pay forward wherever I can, in my mum's spirit.

At times like this the universe can seem cold, and harsh, and unfair; but people like Mum show us that goodness and love abound in this universe. The goodness and love is in us. We get to defy the cold and the unfair. Like mum, each of us gets to choose to be strong, to love, and to make reality wonderful for each other.

Thank you Mum, for caring for me, and Dirk, and Mike, and Dad, and for Orson and Scout, and for always believing in me. Thank you for always laughing out loud at our stupid jokes. Thank you for sitting patiently and learning to code with me on our Apple IIe when I was 8. Thank you for showing me the way to be a good human. I promise I will try.

]]>
/tags/all Fri, 10 May 2019 09:55 GMT
Build a decentralized web chat in 15 minutes entries/build-decentralized-web-chat-15-mins http://mccormick.cx/news/entries/build-decentralized-web-chat-15-mins This post originally appeared on the David Walsh blog.

In this 15 minute tutorial we're going to build a simple decentralized chat application which runs entirely in a web browser.

All you will need is a text editor, a web browser, and a basic knowledge of how to save HTML files and open them in the browser.

Diagram of how WebRTC works browser to browser

We're going to use Bugout, a JavaScript library that takes care of the peer-to-peer networking and cryptography.

Get the full tutorial on GitHub

Read more posts on the subject of cryptography & decentralized systems.

]]>
/tags/all Wed, 27 Mar 2019 13:13 GMT
TOPLAP 15th Birthday Streamed Algorave entries/toplap-15th-birthday-streamed-algorave http://mccormick.cx/news/entries/toplap-15th-birthday-streamed-algorave As part of the TOPLAP 15th Birthday live-stream I live-coded some algorithmic rave music in Speccy:

Speccy is a browser based environment for live-coding 8-bit algorithmic rave music in ClojureScript.

You can watch the videos of everybody who participated here.

]]>
/tags/all Fri, 22 Feb 2019 09:05 GMT
Global Game Jam 2019: Otoch entries/global-game-jam-2019-otoch http://mccormick.cx/news/entries/global-game-jam-2019-otoch album-cover.png

A couple of weekends ago my friend Crispin and I made this game as part of Global Game Jam, an event in which participants build a game in 48 hours.

It was a lot of fun and I got to spend most of the time drawing and doing graphics and music, which was a nice break from writing software.

Play it online!

]]>
/tags/all Mon, 04 Feb 2019 07:39 GMT
Catskills, NY, USA entries/catskills-ny-usa http://mccormick.cx/news/entries/catskills-ny-usa Image1774727300.jpg

So long 2018.

]]>
/tags/all Sat, 29 Dec 2018 16:39 GMT
Inkscape Animation with SVG Animation Assistant entries/inkscape-animation-with-svg-animation-assistant http://mccormick.cx/news/entries/inkscape-animation-with-svg-animation-assistant Update! SVG Flipbook is an app doing flipbook-style layer animation with Inkscape.

SVG Animation Assistant is an open source companion application for Inkscape.

SVG Animation Assistant interface showing Inkscape and a walk cycle animation

It runs along side Inkscape and helps you animate by cycling through the layers of your SVG as you edit it.

This allows you to do basic flip-book style animation. Each layer in your SVG is one frame of the animation.

Customise frame timing and behaviour by editing the layer name:

Inkscape layers UI with customisation

  • Set the number of milliseconds to pause on each frame by entering a number in brackets in the layer name like (100) for a pause of 1/10th of a second.
  • Add static background frames by putting (static) in the layer name.

The animation live-reloads in the assistant window whenever you hit save in Inkscape.

SVG Animation Assistant interface showing live reloading

Run on Linux

If you are a Linux user you can use the online version of this app.

You can get the full source code to this application on GitHub.

]]>
/tags/all Tue, 20 Nov 2018 11:53 GMT
Zero Feature Software entries/zero-feature-software http://mccormick.cx/news/entries/zero-feature-software Let's build software like an axe.

An axe in a block of wood

feature

n.

A prominent or distinctive aspect, quality, or characteristic: a feature of one's personality; a feature of the landscape.

Properties of an axe:

  • Does not have "features".
  • Useful to ordinary people.
  • Does only one thing & does it well.
  • Simple to obtain, use, and maintain.
  • Does not have "upgrades".
  • Works on a platform that nearly everybody has.
  • Is the property of the owner.

For the task of chopping wood the axe is a perfect technology.

An axe is made to do one thing and do it well. It has no features.

An axe is finished. You buy the axe from the store and it chops wood.

The axe design changes only over millenia: slowly and carefully. The axe itself does not change.

An axe does not receive upgrades. Nobody wants "continuous delivery" for their axe. The axe is immutable.

Software axes

Let's build software like an axe.

  • Let's remove all the "features".
  • Let's do one thing and do it well.
  • Let's build things simple enough to secure.
  • Let's finish the software before we release it.
  • Let's release immutable, self-contained artifacts.

By all means, experiment. When it comes time to release let's be disciplined and craft something secure, wholesome, and complete.

Examples of software axes

Hand holding an axe

]]>
/tags/all Mon, 12 Nov 2018 02:36 GMT
Hashcash Auctions for Decentralized Resource Allocation entries/hashcash-auctions-for-decentralized-resource-allocation http://mccormick.cx/news/entries/hashcash-auctions-for-decentralized-resource-allocation Abstract

Hashcash is a mechanism for defending against spam and denial-of-service attacks in email and other decentralized systems. Implementors of systems using hashcash face the issue of how to set the proof-of-work difficulty. A general solution to this problem is given which can be used to predictably allocate resources in decentralized systems where individual nodes each contribute finite resources. Some emergent effects of this solution are explored.

Hashcash client-puzzle token represented as a dollar bill

Hashcash

Nodes in decentralized systems face two closely related issues around the use of their resources. The first problem is that of how to fairly allocate resourcess between connecting clients, and the second problem is that of how to protect themselves against resource depletion attacks. Some systems, like Bittorrent, largely sidestep these issues because the goals of participants (download the data quickly) are aligned. In other systems such as email, nodes are vulnerable to resource depletion attacks like spam and denial-of-service (DoS) attacks.

In 1997 Adam Back proposed an ingenious & practical scheme for countering DoS attacks and spam emails called hashcash. This scheme was famously referenced by, and used with some tweaks, in the software which launched the cryptocurrency revolution: Bitcoin.

As noted in the 2002 Hashcash paper the idea of using a proof-of-work in this way predates Hashcash in the work of Cynthia Dwork & Moni Naor, 1993 and later in the client-puzzles work of Ari Juels & John Brainard, 1999.

Adam Back's implementation is notable because it was based on a modern widely implemented cryptographic hash function, used a cleverly simple leading-zeroes hash collision mechanism which is verifiable with the naked eye, and most importantly of all it came with concise source code which anybody could compile and run ("cypherpunks write code").

The common idea behind these proof-of-work schemes is that you do not allocate resources to the processing or storage of incoming messages unless the sender has proven that they have done a certain amount of computational work. A spammer or DoS attacker is forced to multiply the work required by the number of attacking nodes they are utilising making it more expensive for them to attack. The key to the proof-of-work mechanism is that it is easier for the receiver to verify the proof than it is for the sender to construct the proof, and secondly, that the receiver can specify how much work the sender needs to perform in order to be considered. In Hashcash the construction of proofs-of-work and their more efficient verification is accomplished with a basic building block of cryptographic systems, a computational trapdoor: the hash function. The second key to proof-of-work, that of the receiver specifying the difficulty to require of clients, is not clearly addressed in this body of work. Here we'll describe a general solution to the problem of setting PoW difficulty and explore some emergent effects of the solution on decentralized systems.

Drawing of a pick and shovel

The difficulty of calibrating hashcash difficulty

In Dwork & Naor (1993) they say:

There is a pricing authority who controls the selection of the pricing function and the setting of usage fees.

And later, in the section on the Fiat-Shamir signature scheme:

A reasonable choice is k = 10.

Where k here is equivalent to what we now call the difficulty in cryptocurrency proof-of-work contexts. Basically "here's a reasonable value to use on today's computers". This strategy is not tenable in many decentralized systems because a pricing authority, or "trusted third party" is an inherent security risk (Szabo, 2001).

In Adam Back's 2002 paper there is a section called "Dynamic Throttling":

With interactive hashcash it becomes possible to dynamically adjust the work factor required for the client based on server CPU load.

But it's not clear exactly how much difficulty should be assigned in proportion to the server CPU load.

Futher, on the "Bitcoin" page of hashcash.org:

Hashcash difficulty is static and eroded by Moore's law currently 20 bits.

Again we see a "reasonable value to use on today's computers".

In "How bitcoin uses hashcash":

Hashcash was expected to adjust difficulty every 6 months or year, manually based on observed moore's law estimates. It was proposed that this be measured against a benchmark machine by some vague/undefined human consensus process and broadcast via signed difficulty update headers, and/or long TTL DNS TXT records.

He further points out that Bitcoin's "automatic inflation control" innovation has solved this problem for the cryptocurrency case.

In the 2002 paper again we see the following in the section on mitigation of TCP connection-slot depletion, which concludes with:

Connections will be handed out to users collectively in rough proportion to their CPU resources, and so fairness is CPU resource based (presuming each user is trying to open as many connections as he can) so the result will be biased in favor of clients with fast processors as they can compute more interactive-hashcash challenge-response tokens per second.

The implementation detail of exactly how much difficulty to require per connection slot is again left as an exercise to the reader. Adam Back's original hashcash announcement on the Cypherpunks mailing list makes reference to a fixed difficulty of the first 20 bits of zeroes and as noted in the Wikipedia page on hashcash this becomes a sort of de-facto standard reference value.

Drawing of a flyer with tear-away tabs

Existing solutions

How high to set proof-of-work difficulty is of course not completely vague and unsolved. Bitcoin famously pits miners against each other using a hashcash difficulty that is adjusted every two weeks to a value computed from previously achieved hash difficulties. Referring back to the TCP connection slot depletion example in the 2002 paper, we can think of the Bitcoin miner hashcash use-case to be a slot of size 1 that is only available to be filled every 10 minutes. Hash rates are forced upwards by competition for that single slot between miners who will be rewarded under the incentive model of the Bitcoin system.

There is another mechanism in Bitcoin that we can also borrow from for decentralized systems (absent a fully functional and carefully engineered global store-of-value system called a Blockchain); the fixed block size. In Bitcoin the size of each block storing transaction data, and therefore ultimately the number of transactions, is fixed. This was a point of contention for some time amongst Bitcoin enthusiasts and stakeholders but in the end those supporting a fixed block size had their day and Bitcoin today retains a fixed block size into which transactions from the mempool must be squeezed.

Every Bitcoin transaction is accompanied by a fee paid to the miner who mines the block in which it is included. To decide which transactions should be accepted into the block they are minting miners naturally refer to these fees out of financial self-interest, to ensure they capture the highest aggregate fee possible. When a new block is minted a miner will sort the transactions from those with the highest fee paid to those with the lowest fee paid. Then they will insert into the block the top-N-by-fee transactions that will fit into the fixed block size.

So we have three models to build our solution on: the TCP connection slot example, Bitcoin's automatic inflation control, and Bitcoin's fixed block size.

Consider that in all three of these examples the fundamental resource being allocated is subject to fixed constraint, whether it is TCP slots, Bitcoin issued per 10 minutes, or the transaction-processing capacity of Bitcoin nodes. It is in fact always the case that there are resource limits placed on networked services even if that limit is sometimes so high as to give the illusion of being unlimited. Take for example the original use-case of hashcash, the email inbox: here the limited resource is your own time and attention. There is some practical upper bound on that resource such that you would be fundamentally unable to check your email if you were receiving e.g. 10,000 emails per hour. This is also true of the resources of nodes in any decentralized system.

We can use the fixed block size model in combination with hashcash to balance load and distribute resources efficiently in decentralised systems with automatic difficulty adjustment and without requiring a full cryptocurrency blockchain implementation or trusted central authority.

Drawing of an auctioneer's gavel

Fixed resource hashcash calibration

The "fixed block size" mechanism can be used more generally in decentralized systems to protect nodes from resource depletion and deterministically allocate resources by conducting a "hashcash auction" between clients over the available resources.

The hashcash auction works as follows:

  • Each node specifies the number of clients they can comfortably serve as R, given their resource limits on disk, bandwidth, CPU, etc. This can often be computed based on simple measurement of the resource(s) in question.
  • Nodes issue HMAC'ed tokens ("symmetric key certificates" in the hashcash paper) to clients who request access which include a timestamp so that they can be expired.
  • To use a node's resources, clients must bid for a slot by performing a PoW over these tokens.
  • Nodes publish the upper, lower, and median PoW currently acheived by clients occupying the available resource slots R.
  • Connecting clients are added to the client pool, which is sorted by PoW difficulty-achieved, and only the top R by difficulty are serviced.
  • (Optionally, the lower-bound minimum PoW can be capped, to emulate the same basic DoS protection that hashcash provides.)
  • (Optionally, dependent on use-case, clients are disconnected after some time and required to re-bid for a slot).

In short, you decide how many clients you can service at a time (R), then sort connecting clients by their PoW height, and then deny access to, or disconnect clients who fall outside the top R.

Node awaiting client connections

Note that the server is not required to store any information relating to the the tokens it gives out to prospective clients because it can use the HMAC to verify the tokens it has issued and the time at which they were issued. In this way the issuance of tokens and their expiry becomes very cheap for a service node.

Client bids for a service slot

The natural consequence of a system like this is that clients must compete for available slots to use the node's resources, and a node does not need to specify exactly how much PoW each client must perform. Instead clients arrive at this value through an emergent bottom-up bidding process. The node no longer has to guess at "20 bits" and can instead honestly say "look, here is how hard other clients are working, if you want to access my resources you'll have to do better".

Client 4 out-bids

The clients themselves determine the proof-of-work difficulty setting in an ongoing auction where they must make a higher bid than other clients in order to be allowed access to consume the fixed set of resources, or go elsewhere.

To cope with large numbers of clients and provide a higher level of granularity in the PoW ranking, a simple less-than operator can be used so that clients can bid for intermediate values. For example 0x01fe < 0x01ff so a client bidding with a hash of 0x01fe would beat a client bidding with a hash of 0x01ff.

Properties of this system

What this restricted-resources scheme gives us is a way to measure the sacrifice (Nick Szabo, 2002,2005) that a client is willing to make for the privilege of using a node's resources. As with the human measure of time sacrificed in Szabo's article, it is not an ideal way to measure the importance of incoming client connections, but it is better than existing systems which are essentially a lottery, or in some cases "let the bad guy win". It is not perfect but rather a "proxy measure" which gives a service a deterministic ranking system of roughly those clients who most badly want or need access.

Drawing of an hourglass

At first this system seems unfair - those clients with the most CPU are able to dominate access to resources - but notice that it is much fairer in the worst case where a motivated legitimate client can still get access under DoS or spam attack by working hard to out-compete just a single instance of the attacker. Given enough motivated legitimate users there now also exists a way for them to collectively out-compete a spam or DoS attack. In other words, utilising this scheme has a better failure mode than the free-for-all which normally prevails in networked systems.

Another way of looking at this system is that we have inverted the security aims from "keep out the bad guys" to "keep in the good guys". By giving motivated users a way to express the strength of their need we are able to keep those clients connected who signal the most need to access to the resources. We are able to preference those users who will actually utilise the resource with a higher probability than those users who will waste it, because wasting it is now costly.

A further benefit is that the service utilising this scheme is protected from hard failure. In the worst case failure mode an attacker is only able to deplete resources up to the comfortable hard-limit R specified by the server, meaning the server is never placed under such high load that it entirely crashes and burns, so long as R is specified within serviceable bounds.

In the real world there is always a constraint upon resources, and it is often the case that those wanting to consume a resource must hustle harder to get access. This scheme has an honesty that mirrors the reality of constrained resources rather than pretending that resources are infinite, which is what many historically optimistic decentralized systems do to their detriment. A computer cannot consume more disk space than 100% of disk, and it cannot utilise more CPU time than 100% of CPU time. Modern network services are often deployed without acknowledging these real constraints and with vague hand-waving about "scaling" by adding more machines to accommodate load. That's fine for Silicon Valley backed ventures with money to burn but decentralized systems are often run by volunteers contributing their precious personal resources. Silicon Valley assumptions on volunteer run networks are likely to lead to ruin.

Finally, this scheme has additional emergent benefits in the case where clients have a choice of decentralized service providers. This case is common in decentralized systems where some subset of participants run their own "full nodes" which other clients may then utilize.

Bottom-up load balancing

The further interesting consequence of the hashcash auction scheme arises when we have many service nodes providing the same decentralized service. In the auction-determined proof-of-work difficulty setting, clients now have a clear signal about which nodes are under most heavy use and can freely choose those nodes where the PoW, and hence the resource usage, is lower. Given a choice between node X which requires a very high PoW and node Y which requires a lower PoW, a client will naturally try to connect to node Y first.

Bottom-up load balancing

The result of clients seeking lower-difficulty service nodes is that they spread themselves more evenly across the available servers. This less clustered network graph can lead to a healthier more decentralized system without requiring any kind of central coordination to acheive it. This emergent effect is a sort of bottom-up load balancing system.

Expiry and time preference

Decentralized systems differ in the way in which resources are allocated over time. Sometimes the resource provided by nodes is small and fast, such as UDP packets in Bittorrent's DHT. Other times the resource may be longer lived and more expensive on a per-unit basis, for example disk space storage or a regularly run cron job.

In the situation where resources are long lived and expensive to maintain it makes sense to require clients to periodically re-bid for their access to the resources in question. For example, rather than storing a connecting client's data on disk indefinitely the service node could require that a client re-bids for the disk space once a month or once a year.

A variant on this would be to use a time-based weighting factor on the PoW submitted by connecting clients such that the PoW of more recent connections are weighted higher during bid sorting than older connections. For example if you wanted to preference more recent connections instead of sorting on PoW you could sort on some variant of PoW / (t + 1) where t is the time since connection.

Graph of PoW over t plus 1

In the parlance of economics this introduces a time preference into the system.

Example

Let's close with an example.

Imagine a decentralized social network where users are able to make posts and collect and publish replies to those posts. This can be accomplished in a decentralized way by representing users as key pairs and having every post and every reply signed by its owner's key pair to demonstrate authenticity rather than relying on a centralized trusted third party for authentication. While a user is online they can host their own set of cryptographically signed posts for others to download via a peer-to-peer network, and collect replies to those posts in the same way. However, what happens when the user goes offline, turns off their devices, goes to sleep etc.?

One solution is to have "followers" of the user automatically mirror their posts for other users to download while they are offline. This solution is better than nothing but falls short when a user has few followers or only has followers in the same timezone such that they will all be offline around the same time. Segments of the social network would go dark for periods of time leading to a lack of reliability of access and a form of accidental time-based self-censorship. Some audiences may be satisfied with such a constraint as a tradeoff for a higher degree of privacy and overall censorship resistance, but a mainstream audience is unlikely to accept this lack of reliability.

A more robust solution would be to back this up with a set of volunteer run "full nodes" which users could enlist to mirror data and collect replies in their absence. This architecture immediately begs a number of questions around allocation of the "full node" contributed resources of disk space and bandwidth. How do nodes protect themselves against greedy clients taking all bandwidth and disk space? How do clients protect against spammy replies? How do nodes protect against censorship via resource exhaustion and DoS?

Using the "hashcash auction" we can get a reasonable solution to these problems, bringing higher availability to the system. Those nodes which provide the service of replicating user posts would measure how many clients they are willing and able to serve based on system resources. They can each do this by measuring their disk space for example and allocating a fixed amount of disk per user to a fixed number of users R keeping this constraint within the disk space each has available. Each full node then requires connecting users to bid for a storage slot for the hosting of their posts while offline. Only those clients who are able to demonstrate their need the most strongly will be able to use the offline-replication service of a node. When new nodes are deployed clients will naturally tend towards them as they choose the lower PoW requirement relative to nodes which are serving many existing connections and therefore have higher PoW requirements.

Those users who might run a full node, and others interested in the health of the network, also have a clear signal as to the current load on the whole system and the relative need for more nodes by looking at the PoW they require in aggregate. People who might run a voluneer node are also provided an assurance that there is a fixed cost on the resource they will be donating, making it more likely that people will be willing to run full nodes to support the system. Finally, there is an incentive to running a full node as the user could white-list the cryptographic keys of their own devices and those of their friends and family, meaning that those accounts are always given preference over the accounts of strangers, bringing benefit to both sets of users.

Drawing of Wampum beads

Addenda

  1. In the spirit of "cypherpunks code" I recently published scrypt-hashcash which provides a basis upon which to validate some of these ideas on the web. For example browser based clients could bid for resources by performing a hashcash PoW using this library and submit it via HTTP, WebSocket, or WebRTC connection to the service conducting the resource auction.

  2. It is interesting that Hal Finney touched on a related idea in RPOW enabled BitTorrent and in true cypherpunk tradition offered code implementing his ideas:

Imagine a version of BitTorrent where nodes that have completed their downloading (called "seeders") could earn RPOWs by continuing to upload. And what could they do with those RPOWs? Imagine further that this version of BitTorrent allowed nodes to send RPOWs in order to get higher priority from other nodes which are downloading. Just such an experimental, RPOW-enhanced version of BitTorrent is now available for download.

Read more posts on the subject of cryptography & decentralized systems.

]]>
/tags/all Sat, 27 Oct 2018 07:11 GMT
Schiphol 23: Airport Missions MMOG entries/schipol-23-airport-missions-mmog http://mccormick.cx/news/entries/schipol-23-airport-missions-mmog Schiphol 23 mockup

I had an idea for a video game a while back. It's a multiplayer mission game with rogue-like elements, set in various procedurally generated airports through which you can transit.

Airport ambience by wichiogarcia on freesound.org

Everybody's missions are all mixed up together, so you might have a mission to deliver a package and somebody else's mission is to stop you. You might be trying to transit throuh several airports and other people are trying to catch you. You might be trying to find an item that somebody else has hidden. You might be chaperone to a VIP and the VIP is another player. Missions would last an average of 5 minutes each and feature a count-down timer as in the game Counterstrike.

The visual style would be vector based similar to those isometric maps you sometimes see in airports.

Amsterdam airport map

Brussels airport map

I don't play many video games but I very much enjoyed the pace and aesthetic of the game Rymdkapsel by Martin Jonasson.

Rymdkapsel screenshot

I found it was an easy game to put down and pick up again for short bursts of play.

Probably my favourite video game is Another World by Eric Chahi which features a vector style and is similarly easy to play in small increments.

Another world

]]>
/tags/all Sun, 07 Oct 2018 00:07 GMT
Solderless Prototyping entries/solderless-prototyping http://mccormick.cx/news/entries/solderless-prototyping DSC_2597.JPG

]]>
/tags/all Wed, 03 Oct 2018 13:56 GMT
Too Good to be True entries/too-good-to-be-true http://mccormick.cx/news/entries/too-good-to-be-true DSC_2618.JPG

]]>
/tags/all Thu, 27 Sep 2018 09:07 GMT
Decentralized Identity Linking entries/decentralized-identity-linking http://mccormick.cx/news/entries/decentralized-identity-linking Decentralized identity linking animation

A problem faced by decentralized systems is that of naming things. The problem is best expressed by Zooko's Triangle which conjectured that no single kind of naming system can provide names satisfying all three of the following properties:

  • Human-meaningful: Meaningful and memorable (low-entropy) names are provided to the users.
  • Secure: The amount of damage a malicious entity can inflict on the system should be as low as possible.
  • Decentralized: Names correctly resolve to their respective entities without the use of a central authority or service.

Something as simple as a user handle or nickname becomes a complex issue in a decentralized system. What you ultimately want is something like a decentralized Twitter handle. The handle is a single human-memorable string which points to one and only one user in the system. The first user to register a particular name gets to claim that name. Obviously the case of Twitter handles is not decentralized as it requires the Twitter database as a single point of authority on who owns which name.

As the Wikipedia article points out, and Zooko has acknowledged, the invention of the Blockchain allows for solutions under certain conditions. However the effort required to design, build, and deploy a Blockchain along side your distributed application are quite steep. If you want to use an existing Blockchain implementation then either your users must be invested in it or your app must invest in it and pay for name registrations.

The former situation requires your users to not just install your app but also to acquire some particular Blockchain currency which steepens the barrier to entry for your app. The latter alternative requires your app to fork out for registrations which means the naming system of your decentralized app will fail if you or whichever entity is paying for the names shuts down.

The Keybase Method

In a decentralized system "one user" often means "one or more cryptographic key pairs". The site Keybase offers a good alternative naming system in the situation where "user" means "keypair". The solution is to use existing online identities held at different providers and link them cryptographically to the user's key(s). This means that third parties can verify an identity as follows:

@zooko on Twitter has public key B123h9EAgVdnXWjxUa3N9Amg1nmMeG2u7.

Or to put it more rigorously:

A person who has control of the private key corresponding to public key X also has write access to online service at Y.

A lot of the time when we want a name for something, the latter is what we actually want. We want to know that the existing relationship and trust I have for @zooko on Twitter can be carried over into this new decentralized system I have started using. If I have multiple existing relationships on different services with @zooko on Twitter then I can also verify those and in the decentralized app I can give @zooko the simple name "zooko" (which can often be inferred from usernames on services) and refer to the person with that handle from that point forward.

The way that Keybase does this is to help the user to sign a statement with their signing key saying "I am @name on service X" and then post the text and resulting signature on the service itself and keep a record of where somebody can find the post. If you do this it means you can prove to people that your signing key is associated with some existing internet identity such as your Twitter account. People who want to remain anonymous or pseudonymous with respect to their existing internet identities can simply skip this linking step.

This technique can actually be used to link any internet location that the user has write-access to back to a public key that they have the private key for. You simply sign a statement saying "I have write access to location X." and post it at "location X". Some examples:

  • I have write access to https://mccormick.cx/ -> post signed message as textfile there.
  • I have write access to https://twitter.com/mccrmx -> post signed message as tweet there.
  • I have write access to https://github.com/chr15m -> post signed message as gist there.

Links to the signed post are kept and shared with other users who wish to verify the existing relationship.

A hand drawn lock

The decentralized "Keybase method"

Keybase is a centralized website and system which means that it has convenient integrations and is easy to use, but it relies on their infrastructure and centralized, proprietary software. The good news is it's possible to use the exact same technique that they use in a decentralized system. Doing this gives a user the power to make statements like:

@zooko on Twitter is me (B123h9EAgVdnXWjxUa3N9Amg1nmMeG2u7) on this decentralized service.

And then other people can confidently start using the name "zooko:twitter" in the place of B123h9EAgVdnXWjxUa3N9Amg1nmMeG2u7 in the decentralized system. In fact if the person has signed proof-of-write-access on multiple services and they have the same username on those services we might as well just start calling them "zooko" instead of "zooko:twitter+github". If two zooko's come along then we can just refer to them by their specific service-postfixed names, or some other qualifier, so that we can differentiate when interacting with them - see below for more on solving this "two Johns" problem.

What this means is that "zooko" in the decentralized system becomes a short-hand way of referring to an entity with whom we interact on a constellation of sites. In some ways this is more robust than naming systems on centralized systems (where anybody can "steal" a username by registering it before somebody else) because the alias actually represents the relationship rather than an entry in one specific proprietary database.

So how can we do the same thing in a decentralized system? Quite easily:

For each service the user is on:

  • They sign a statement saying they have write-access to their account at that proprietary service with the signing key that they use on the decentralized service.
  • They then post the signed statement to the proprietary service.
  • When users are connected for the first time each user shares their links to "proof of linked identities" (if any) so that they can verify any existing Internet identities.

No-namers & the contacts list model

What about in the case where somebody does not sign anything to say that they own some existing online aliases? What if your Mum starts using decentralized service X and she does not have a GitHub or Twitter account? What if somebody chooses not to link their existing identities and instead decides to start afresh?

In the real world we don't really get to name ourselves. We can suggest a name to people, but they get to call us whatever they want. This is how nicknames happen and it's a bit of an aberration that in the online world we get to choose one canonical name by which people call us on proprietary services.

What if naming in online systems worked more like the decentralized real world? What if we suggested a name by which somebody could call us, but it was up to them to use it or to choose a different name by which to call us? What if, in this new decentralized service that my Mum has started using, I simply refer to her as "mum"?

If somebody has no existing online identities they can simply suggest a name when two users first connect and trade keypairs. The receiving party can then choose to adopt the suggested handle for that user, or if there is some conflict in their existing set of collected aliases, add a qualifier, or choose an entirely different name that makes sense to them.

This is the "contacts list" model of naming. You can tell me your name but I don't have to use it for your entry in my contacts list. I can adopt the name you've suggested but if I know two "Johns" then I'm likely to add some other qualifier like "Tennis John" and "Football John".

Aside: it would be funny if even DNS worked like this. The first time you loaded Google it would suggest to you "please call me google.com" but you could choose whatever name you wanted to type into the URL bar from that point forward. Norms would form between humans using the internet for what different sites would be called, and sometimes they would be called something different than what the person owning the domain wanted.

I don't think I would use "google.com" for example. I can think of some better names for that site.

Summary

The system proposed here for naming users in decentralized way uses an amalgam of two models: Keybase-style service signing, and the contacts list model.

Detailed view of decentralized identity linking contact card

To summarize the process of managing identities in this system:

  • Each user in the decentralized app is identified by one or more cryptographic key pairs.
  • A user may set a suggested handle they'd like other people to use.
  • Optionally a user is able to add existing identities (Twitter, GitHub).
  • Optionally the app helps the user create cryptographically signed back-links from their existing internet identities to the keypair.
  • The app stores the user's links to their signed posts if any.
  • When user A sees some user B for the first time the client receives the following information about user B:
  • Their public key or fingerprint.
  • Their suggested handle.
  • URLs of their cryptographically signed identity-linking posts.
  • The app then fetches user B's identity-link posts and verifies the signatures found there.
  • Optionally user A is able to modify or supply their own local name (handle) for user B.
  • The app creates a contact entry for user B with this information.
  • When displaying user B the application uses the handle (chosen by user A) stored in the contact card.
  • Under the hood the application uses user B's public key or cryptographic fingerprint/address to refer to that user.
  • In the user interface the contact card is visible appropriately: for example on hover of user B's handle.
  • The displayed card shows user B's handle, identity-links with verification status, and public key or fingerprint.

The raw public key or fingerprint(s) can additionally be used to verify real world identity out-of-band such as at key signing parties, personal meeting, or over a known secure channel.

Read more posts on the subject of cryptography.

]]>
/tags/all Fri, 31 Aug 2018 01:46 GMT
Lost Worlds entries/lost-worlds http://mccormick.cx/news/entries/lost-worlds lost-worlds-1.jpg lost-worlds-3.jpg lost-worlds-2.jpg lost-worlds-4.jpg lost-worlds-5.jpg lost-worlds-6.jpg lost-worlds-7.jpg lost-worlds-8.jpg

Recent-ish sketches. Been learning from The Etherington Brothers, Dr. Seuss, and How to Build Treehouses, Huts, and Forts.

]]>
/tags/all Thu, 23 Aug 2018 05:54 GMT
On Self-hosting and Decentralized Software entries/on-self-hosting-and-decentralized-software http://mccormick.cx/news/entries/on-self-hosting-and-decentralized-software Sketch of a tree

Web applications often follow a client-server model meaning that there is a piece of software which runs in your web browser (the client) and a piece of software which runs on a server somewhere. I'm interested in this model, where it came from, and where it's going, and I discuss this below. I'll also discuss a new model for self-hosting web applications that I've been exploring.

tl;dr

  • To "self-host" a web application means to run the client and server on computers you own.
  • Self-hosting is a basic foundation of decentralization.
  • The "installable app" model on phones & PCs is quite decentralized.
  • Decentralization is robust and anti-fragile.
  • How can we bring self-hosting of web apps to everyone?

Brief history of software applications

In the old days most software people ran was in the form of applications that you would download onto your PC and run. This was quite a decentralised model in that you could find whatever software you liked and run it on your machine without asking permission from anybody.

The modern mobile-device app stores and the web have deviated from this model. In the case of app stores you can only download software that is sanctioned by the company who runs the app store. On Android it is quite easy to change the settings on your device to install software applications from any source but this is not the default, and on Apple devices it's very difficult to do.

The web has a more complicated approach. Running software in your web browser is generally as easy as visiting a URL and people are free to run whatever websites and "web apps" they like on their devices. As mentioned above however, the server-side portion of the web application you are using is often running on somebody else's server computer and not in your control.

Client-server model on the web

Over time more and more functionality has moved from server side code into the web browser. Applications like Gmail pioneered this transition. Recent browser technologies like WebRTC and service workers have given a stronger base to build client-only applications on. The "add to homescreen" paradigm, whilst relatively obscure, presents a vector for running web applications on your computers and devices in a similar way to installed applications.

The buzz-word "serverless" captures the zeitgeist: moving more and more of the software off the server and into the web browser client. The reason software developers are generally in favour of this move is because it is easier to deploy code to web browsers than it is to deploy code to servers; it's easier to get users to run that software (just send them a link); and there is less of a maintenance burden if you don't have to keep a server running.

There's another strong reason why client side code is more favourable than server side code and that is because most server code is running on servers you don't own. For example when you run Gmail the server code and all of your emails are completely controlled by a third party and trusted third parties are security holes.

Even "serverless" in the context of web apps just means that the server component requires less infrastructure and is easier for developers to deploy. The code is still running on a machine owned by some third party who is neither the user or the developer.

Whilst this move towards client-side web applications is good in terms of user control, a problem even with "serverless" web applications that run in your browser is that the software can be modified without your consent. When you load up a web application tomorrow the developer might have made a new version with features you don't like and your browser will load the new version automatically even if you wanted the old version. Even worse, because you load a fresh copy of the software each time you use it, a man-in-the-middle attacker might sneak a malicious bit of code into the version you load tomorrow that is not there today.

There are also many use-cases where only a piece of server-side software will do. Generally these use-cases fall under the umbrella of asynchronous communication, such as storing a file for later retrieval, sending a message to one or more people which might only be received when you are offline, syncing state between devices which might be connected at different times etc.

Examples of things the server code will typically take care of:

  • Connecting client apps together in real-time.
  • Data manipulation functions.
  • Proxying network requests.
  • Persistent data storage.
  • Running periodic jobs.
  • Access control.

So the world is trending towards a more fragile situation of centralized control of the software we all rely on. It would be better for humanity if we were instead moving towards decentralized models of software where an individual can run and control the software they want without fear of a fragile centralized system failing or being used against them by powerful interests and hackers.

Sketch of a tree

Self-hosting

So these latter problems are some of the reasons why self-hosting web applications is still a good move in terms of security and control. When self-hosting, all of the software and all of the data is running on your own computers and you don't have to trust any third party to not be evil, corrupt, make mistakes etc. You can choose which software to install and run and when without obstruction. You get the benefits of cloud connected software - shared state between devices and other users - without having to trust a third party.

The main problem with the self-hosting approach is that it is very technical and therefore simply not an option for most users. Some of the barriers to people self-hosting their web software stacks:

  • Renting or building a server.
  • Installation of server side code.
  • SSL certificate management.
  • Server maintenance.
  • Access control management.
  • Securing the server.

These are the types of things that systems administrators are paid a lot of money to take care of because they are difficult and skillful work. These are the things that big companies are doing for you when you use their online services. The unwritten contract we engage in today is that we give big companies control over our digital life and our valuable personal data in exchange for their building, installing, and maintaining the infrastructure code on their servers.

An additional problem is that even in the "host your own server" model there is a lot of centralization and passing around of valuable and private personal information. For example obtaining a domain name and SSL certificate requires central authorities who almost always require a lot of personal information. Not to mention forking over cash. Additionally, even if you host your code on a VPS - as is the case with almost all server side code on the internet - then some trusted 3rd party has unlimited access to the machine you are renting.

Sketch of a tree

Some existing proposed solutions

So the interesting question is whether there is some way to make self-hosting possible for everybody in the same way that PCs and phones made "run the software I want" easy for everybody.

There was a time when a lot of web applications were deployed as simple PHP scripts with HTML, CSS & Javascript for the front-end. This was probably the simplest format for self-hosted applications as it was easy to find hosting providers who would let you run PHP, and installation was as simple as copying files up to a server. I think this is a big reason why PHP has remained popular for so long despite the fact that it's such an awful language with frequent security issues. It simply let you do what you want. These days though, a lot of modern software on the internet is not written in PHP and developers often prefer other langauge ecosystems.

Another idea for self-hosting was the Freedom Box pioneered by Free Software enthusiasts in the 2000s. The idea is pretty good but does not seem to have acheived widespread adoption. My guess is the use cases are still too obscure for most users. what does "run your own XMPP server" mean? Most people have no idea.

A lot of people today are of the mind that "blockchain" is the singular solution to the issue of decentralizing the software we run. In this model you pay in small increments for the server side code which runs on a distributed system shared amongst many participants. You maintain control of your piece of the distributed service and your data stored in it using cryptography. In this model the server component and app state is shared across many computers all at once and you basically pay for a small slice of it.

This is a fascinating idea but I am not completely convinced that "blockchain" will be the future of decentralized software applications because:

  • So far it is unproven in all except one application, that of decentralized digital gold.
  • There are a number of very successful decentralized systems that function well without any kind of token/monetary incentive (email, irc, http, bittorrent, git etc.) and that original model of decentralization by cooperating parties is still a sound, well proven one. People still use email.
  • The blockchain design encapsulates a trade-off of low efficiency for immense security and trust minimization. This means that blockchains are very good at storing and securing transaction data but they are not very efficient at storing lots of other types of data. Imagine having your own dropbox folders also filled with encrypted copies of those of everybody else in the world. It does not scale.

In addition to the above ideas there exist new ecosystems like ownCloud and Sandstorm which enable people to deploy their own "stacks" hosted on their own servers. These are noble efforts at decentralization and self-hosting into which many people have poured a lot of work. The problem with these "host-it-yourself cloud stacks" is that they often still require trust in some 3rd party hosting service, and quite deep technical knowledge. Sure, to a nerd it is easy to upload a PHP script and configure a MySQL server, purchase VPS hosting, install a Let's Encrypt SSL certificate, but it is not always so easy to a nerd's mother, father or friend.

Sketch of a tree

A simple platform for self-hosting

I've been thinking about all of this in recent years. I've been asking myself these questions:

What if running persistent server side code was available to anybody, even non-technical users? What if installing server side software was as click-tap easy as it is on a phone or PC? What if you could run cloud software and servers on the desktop PC in your office, or tablet, or Raspberry Pi on your home network? What does it look like when anybody in the world can self-host their "cloud" software?

Criteria for such a system would be:

  • Easy to set up and administer.
  • Easy to install and uninstall new services.
  • Interfaces easily with web browser client software.
  • Secure.
  • Completely user-controlled and decentralized.

As I've thought about these ideas I've also been tinkering with WebTorrent. It's a cool piece of technology which is bringing a chunk of the BitTorrent architecture to the web.

Working with this technology made me realise something the other day: it's now possible to host back-end services, or "servers" inside browser tabs.

Client-server model with WebRTC

Instead of a VPS server running in some data center, you have a browser tab running on a spare computer. Instead of clients talking to servers over HTTP they can talk over WebRTC. You build your "backend" service code with the same tech as your client side web app, with HTML and Javascript.

When I realised this it blew my mind. I couldn't stop thinking about it. Maybe the persistent browser tab on the home PC can be the new server for self-hosting users? What if grandma could build out a server as easily as opening a browser tab and leaving it running?

So anyway, I've made this weird thing to enable developers to build "backend" services which run in browser tabs. It's called Bugout and it has been fun to hack together.

Check it out and let me know what you think.

]]>
/tags/all Sun, 12 Aug 2018 12:38 GMT
Live-code 8-bit algorave music in the browser with Clojurescript entries/live-code-8-bit-algorave-music-in-browser-with-cljs http://mccormick.cx/news/entries/live-code-8-bit-algorave-music-in-browser-with-cljs

Speccy is a small utililty I built for live-coding chiptune music in the browser with Clojurescript.

You can copy sounds from sfxr.me and paste them in as synth definitions, using code to modify any parameter over time, or start from scratch by building a synth up from basic parameters.

You can paste the following examples into the online editor to try it out:

; blippy synth
(sfxr {:wave :square :env/decay 0.1 :note #(at % 32 {0 24 3 29 7 60 12 52 19 29 28 52})})

; donk bass
(sfxr "1111128F2i1nMgXwxZ1HMniZX45ZzoZaM9WBtcQMiZDBbD7rvq6mBCATySSmW7xJabfyy9xfh2aeeB1JPr4b7vKfXcZDbWJ7aMPbg45gBKUxMijaTNnvb2pw"
      {:note #(or
        (at % 57
          {5 35
           27 34})
        (at % 32
           {0 24
            6 29
            18 21
            26 12}))})

; hi hat
(sfxr {:wave :noise
       :env/sustain 0.05
       :env/decay 0.05
       :vol #(sq % [0.3 0.1 0.2 0.1])})

; snare
(sfxr "7BMHBGCKUHWi1mbucW62sVAYvTeotkd4qSZKy91kof8rASWsAx1ioV4EjrXb9AHhuKEprWr2D4u4YHJpYEzWrJd8iitvr23br2DCGu7zMqFmPyoSFtUEqiM64"
      {:note 36
       :vol #(at % 16
         {4 0.5
          12 0.5})})

The full source code and documentation is available at GitHub.

Enjoy!

]]>
/tags/all Fri, 03 Aug 2018 02:14 GMT
Hash Function Attacks Illustrated entries/hash-function-attacks-illustrated http://mccormick.cx/news/entries/hash-function-attacks-illustrated Here are some illustrated explanations of the main ways in which cryptographic hash functions can be attacked, and be resistant to those attacks.

Zooko Wilcox's blog post Lessons From The History Of Attacks On Secure Hash Functions gives us a nice overview of these and I've quoted his concise explanations below. Check out his great post for more detail and history on this topic.

A cryptographic hash function is an important building block in the cryptographic systems that keep us safe in our communications on the internet.

A hash function takes some input data and generates a hopefully unique string of bits for each different input. The same input always generates the same result.

Diagram of an ideal hash function

The input to a secure hash function is called the pre-image and the output is called the image.

I use the following key below:

A red square

Red for inputs which can be varied by an attacker.

A green square

Green for inputs which can't be varied under the attack model.

To attack a hash function the variable inputs are generally iterated on in a random or semi-random brute-force manner.

Collision attack

Diagram of a collision attack on a hash function

A hash function collision is two different inputs (pre-images) which result in the same output. A hash function is collision-resistant if an adversary can't find any collision.

Pre-image attack

Diagram of a pre-image attack on a hash function

A hash function is pre-image resistant if, given an output (image), an adversary can't find any input (pre-image) which results in that output.

Second-pre-image attack

Diagram of a pre-image attack on a hash function

A hash function is second-pre-image resistant if, given one pre-image, an adversary can't find any other pre-image which results in the same image.

Hopefully these diagrams help to clarify how these attacks work!

Read more of my posts on the subject of cryptography.

]]>
/tags/all Mon, 30 Jul 2018 02:14 GMT
Fridgetown & Donnelly River July 2018 entries/fridgetown-amp-donnelly-river-july-2018 http://mccormick.cx/news/entries/fridgetown-amp-donnelly-river-july-2018 DSC_1961.JPG DSC_1962.JPG DSC_1963.JPG DSC_1969.JPG DSC_1970.JPG DSC_1982.JPG DSC_1988.JPG DSC_1989.JPG DSC_2005.JPG DSC_2008.JPG DSC_2009.JPG DSC_2011.JPG DSC_2024.JPG DSC_2029.JPG DSC_2032.JPG DSC_2040.JPG

]]>
/tags/all Mon, 16 Jul 2018 12:52 GMT
Talk: GNU/Linux in Tiny Places entries/talk-gnu-linux-in-tiny-places http://mccormick.cx/news/entries/talk-gnu-linux-in-tiny-places gnu-linux-in-tiny-places.jpg

Last night I gave a talk at a Perth Linux User's Group meetup about doing Linuxy stuff on small machines:

]]>
/tags/all Wed, 11 Jul 2018 08:10 GMT
Night Mushrooms entries/night-mushrooms http://mccormick.cx/news/entries/night-mushrooms DSC_1943.JPG

]]>
/tags/all Sat, 07 Jul 2018 00:52 GMT
Frock: Clojure-flavoured PHP entries/frock-clojure-like-php http://mccormick.cx/news/entries/frock-clojure-like-php Frock is a little experimental tool for writing PHP scripts using Clojure-like LISP syntax.

If you want to see what the code looks like, here's an example which fetches and lists top news items from the Hacker News API.

Some Frock code

Frock could be interesting to you if you are LISP or Clojure programmer writing a web application which is mostly front-end code, but which needs some small amount of server side logic for e.g. proxying, authentication, data persistence etc. and you want this application to be easily deployable by semi-technical users on commodity hosting.

Basically if your target audience is graphic designers, you like Clojure, and your backend requirements are slim, then you might be interested.

Why?

Pythagoras says no to Fava beans

PHP is an old server-side web development language which is simultaneously loathed by software developers everywhere, and also wildly popular and widely deployed. To reconcile this paradox let's take a look at some pros and cons of PHP.

Cons:

  • Ugly language semantics & features.
  • Dubious security record.
  • Much awful legacy code lying around.

Pros:

  • User-friendly app deployments (simply copy files to server).
  • Widely available on internet servers.
  • Mature language and ecosystem.
  • Excellent documentation.
  • Much useful tech bundled ("batteries included").

The pros make PHP quite democratic. It's very easy to install PHP code on widely available, cheap, commodity hosting. It's easy to get started writing PHP applications; the PHP binary comes pre-installed on OSX for example. PHP contains a lot of capabilities by default: zipping files, opening sockets, encryption, command execution.

Frock exists to make the language semantics and features less of a con for brace wrangling LISP heads, whilst retaining the wide deployment surface and other democratic features of PHP.

]]>
/tags/all Wed, 27 Jun 2018 08:51 GMT
Browser Blockchain in ClojureScript entries/browser-blockchain-in-clojurescript http://mccormick.cx/news/entries/browser-blockchain-in-clojurescript I built a little blockchain-in-a-browser in ClojureScript to help understand the underlying algorithms.

You can simulate a network of peers by opening multiple browser tabs. Each peer can mine blocks and make transactions independently and the resulting blockchain will resolve conflicts correctly across all tabs.

A blockchain works by laying down a chain of blocks of transaction data.

Bitcoin whitepaper SPV

Each block in the chain contains a cryptographic hash with two important properties:

  • It proves a link to the previous block.
  • It proves that difficult computational work has been done.

The proof-of-work is accomplished by iteratively updating a nonce until a low-probability hash is discovered.

These two properties mean a blockchain is digital amber.

Insect embedded in amber

If somebody wants to modify a transaction deep inside the amber it would be very difficult because they would have to re-create every layer of the blockchain by doing as much work as the original process required.

In my browser blockchain the hashing is implemented like this:

(hash-object [timestamp transactions previous-hash nonce])

As you can see the previous block's hash is included in the current block.

The hashing is performed iteratively in a loop until a hash with at least one byte of leading zeroes is found:

(loop [c 0]
  (let [candidate-block (make-block (now) transactions previous-hash new-index (make-nonce))]
    (if (not= (aget (candidate-block :hash) 0) 0)
      (recur (inc c))
      candidate-block)))
]]>
/tags/all Wed, 13 Jun 2018 01:30 GMT
Across the Sea of Space entries/across-the-sea-of-space http://mccormick.cx/news/entries/across-the-sea-of-space Mountain with space ship

letters.png

helmet.jpg

eyes.png

Romanesque arial

ship-and-planet.jpg

Hillside, tree, and spacecraft

computer-tunnel.jpg

Airballoon over a valley

]]>
/tags/all Wed, 09 May 2018 05:22 GMT
Scitech Show entries/scitech-show http://mccormick.cx/news/entries/scitech-show DSC_1197.JPG

I'm playing a show tonight at the local science education centre, Scitech, for their adult event, After Dark.

I'll play live electronic music using some Gameboy, Commodore 64, and Raspberry Pi gear and algorave Pd patches. I'll also give a little talk about the tech and how it works.

Scitech, After Dark

]]>
/tags/all Sat, 14 Apr 2018 00:59 GMT
Plastimake's Great for Making Homethings entries/plastimake-s-great-for-making-homethings http://mccormick.cx/news/entries/plastimake-s-great-for-making-homethings plastimake-models.jpg

A couple of years ago my friend Joe recommended this stuff called Plastimake to me.

It is a hard plastic which goes soft and malleable at around 60 degrees celcius. You drop some in boiling water until it goes clear and soft and then you can mold it into whatever shape you want. It cools and hardens again quickly in whatever shape you have molded it into. You can re-heat it to soften and re-use it again.

Scout and I have used it several times to hand craft little figures for play. I've also used it around the home and when prototyping things as it's a very quick way to get a hard plastic into exactly the shape you want.

Here's their video which is clear and honest:

]]>
/tags/all Thu, 01 Mar 2018 04:19 GMT
Internet of Things Questions entries/internet-of-things-questions http://mccormick.cx/news/entries/internet-of-things-questions A list of real questions that could be answered by teensy little technology nuggets.

Raspberry Pi with Waveshare Rpi Hat

  • In which physical folder is my child's birth certificate?
  • What items of mine are not here?
  • How much hand soap refill is left in my bathroom cupboard?
  • What is the smallest storage space in which I can fit all of my furniture?
  • Which items that I own do I never use?
  • At what time of day do I interact with different items I own?
  • What is the total weight of my filing cabinet and work desk?
  • I need a new one of these, pull up the page where I purchased it in my browser.
  • This thing does not belong to my grandmother. Who did she borrow it from before she died?
  • List the items in my craft box which have "needle" in the name.
  • How much television are my kids watching and what percentage are educational shows?
  • Are there any NE5532s in my tool box?
  • Graph my family's diet composition. How much of it is fresh food?
  • Place a grocery order under $50 for the items I most commonly use that I will soon run out of.
  • Search all of the physical books in my personal library for the following phrase.
  • Are my running shoes in this house or the holiday house?
  • We are leaving this hotel. Have we packed everything?
  • Where have I put my phone and keys?

Generic USB Sound Device

Most of these question could be answered without an internet connection. "Of Things" sounds kind of weird by itself though.

"Internet connected lightbulb" does not provide a use-case that anybody wants or needs.

The things should be telling us their data. They should not be telling corporations our data.

What you want is dumb little things that do one thing well. Good traditional design plus CPU, I/O, and memory.

Things that talk straight to us [and nobody else] with IR, sound, light, QR code, and raw radio.

Is there a word for "Internet of Things" objects which don't have an internet connection?

Maybe "ov things" lol.

"Neat little ov thing you got there friend."

]]>
/tags/all Fri, 16 Feb 2018 12:22 GMT
Tiny Core Linux on Raspberry Pi entries/tiny-core-linux-on-raspberry-pi http://mccormick.cx/news/entries/tiny-core-linux-on-raspberry-pi Tiny Core Linux SSH screenshot

I'm discovering Tiny Core Linux, a beautifully executed, minimalist, immutable-by-default GNU/Linux distribution that runs nicely on the Raspberry Pi and other x86 and ARM devices.

The documentation is comprehensive but arcane so here are my field notes.

I'm using piCore the Raspberry Pi variant of TinyCore but most of these notes should work for other variants too. These notes assume you are already a GNU/Linux desktop user and know basic Linux command line fu.

Happy penguin jumping

I'm a huge fan of this distribution!

Official disk images.

Here's the official README for the Raspberry Pi version.

Write the disk image to your SD card

sudo dd if=piCore-9.0.3.img of=/dev/mmcblk0

Make sure you get your sd card's device (e.g. mmcblk0) correct so you don't overwrite the wrong disk!

Tip: you can see where dd is up to by sending it a signal:

sudo kill -SIGUSR1 `pgrep -f mmcblk0`

After this put the sd card into your Raspberry Pi and it'll boot Tiny Core Linux with an SSH server running so that you can access it remotely over the network.

Find the Pi on your network

find-pis: use this little script to find the Raspberry Pi on your network so that you can ssh to it.

Default login:

username: tc
password: piCore

Immutable by default

Tiny Core Linux is a perfect fit for Raspberry Pi because it does not write to disk by default. Note that between reboots if you want to persist any changes you've made, you need to do it manually with filetool.sh -b. You'll especially want to do this after first boot to persist the SSH server keys. This includes any changes you make to files in the home directory.

This lack of disk writes is a huge boon because it prevents issues that distributions like Raspbian have with SD card corruption when you power off the Raspberry Pi. Immutability is also just a good thing generally for building robust software systems.

Tiny Core Linux package repository

The packages are .tcz files which like most package formats are a zipped up collection of files with some metadata. You can browse the packages available to the Raspberry Pi distribution.

Various useful commands

Persist current state to disk:

filetool.sh -b

Where to specify additional default file locations for persistence:

/opt/.filetool.lst

Download and install packages (nodejs binary package in this example):

tce-load -wi node.tcz

Show which packages are currently installed:

tce-status -i

Install a package for doing C/C++ compilation & development:

tce-load -wi compiletc

Find current OS version info:

version

Search for and install packages with console UI:

tce

Where to put your own startup and shutdown scripts:

/opt/bootlocal.sh
/opt/shutdown.sh

Getting WiFi working

You'll want the packages wifi and firmware-rpi3-wireless and then reboot and set up a connection:

sudo wifi.sh

Don't forget to persist your changes.

Getting sound working

You'll want the alsa and alsa-utils packages.

I managed to compile Pure Data and output a basic test tone. I went into the src folder of Pd and issued make -f makefile.gnu to build it from the source after checking it out from git.

GPIO access

I managed to interface with the Raspberry Pi GPIO by installing node and then using the onoff package installed via npm. Lots of dev packages were required for a successful build (compiletc probably covers most of them). Each time npm threw an error I looked at what the error message was to determine which package to install next until it built successfully.

I managed to get an LED to blink on pin 16 with this code:

var Gpio = require('onoff').Gpio;
led = new Gpio(16, 'out');
led.writeSync(1);
c = 0
setInterval(function() { led.writeSync(Number(c = !c)); }, 1000);

I had to run node as root to get access to the GPIO.

Philosophical waxing

GNU/Linux has now been subsumed into the technological substrates of the world. It is found in most phones, servers and appliances that people interact with every day. There are probably 10 copies of Linux running in the median developed-world household. It has become infrastructure, like roads and wires and water pipes.

I don't see that many young people at GNU/Linux meetups these days. I don't think we GNU/Linux nerds are required by the world in the same way that we were before when it was sparkly and new.

It's exciting for me to find Tiny Core Linux which has re-ignited the spark of enthusiasm for this technology.

]]>
/tags/all Sat, 03 Feb 2018 05:34 GMT
Tech Nugget entries/tech-nugget http://mccormick.cx/news/entries/tech-nugget DSC_0413.JPG

Secret project.

]]>
/tags/all Mon, 29 Jan 2018 04:35 GMT
The 2018 Situation entries/the-2018-situation http://mccormick.cx/news/entries/the-2018-situation DSC_0149.JPG

]]>
/tags/all Mon, 01 Jan 2018 01:46 GMT
Other Worlds entries/other-worlds http://mccormick.cx/news/entries/other-worlds planet-0.png planet-1.png planet-2.png planet-3.png planet-4.png planet-5.png

I just finished another sketchbook.

Recently a friend linked me to The Etherington Brothers and I've been doing their tutorials.

Lately I've been concentrating on plants.

]]>
/tags/all Fri, 24 Nov 2017 12:49 GMT
Technological Artifact entries/technological-artifact http://mccormick.cx/news/entries/technological-artifact Hand drawn technological artifact in the style of Scott Robertson

In the style of this guy in the style of Scott Robertson space ship sketches.

]]>
/tags/all Tue, 17 Oct 2017 03:55 GMT
Lovelace Day 2017 entries/lovelace-day-2017 http://mccormick.cx/news/entries/lovelace-day-2017 A non-exhaustive list of technologies that I used this year which happen to be built by women:

I feel grateful to these people for enriching technology with their contributions.

]]>
/tags/all Wed, 11 Oct 2017 06:58 GMT
The Mountains entries/the-mountains http://mccormick.cx/news/entries/the-mountains IMG_20170912_170725_387.JPG

Some mountains I copied from some website.

]]>
/tags/all Tue, 12 Sep 2017 09:09 GMT
Cloud Notepad entries/cloud-notepad http://mccormick.cx/news/entries/cloud-notepad Cloud Notepad on the Desktop

I love this piece of software and I use it every day. It is a simple note taking application which syncs across all of your devices. It is a self-hosted single-page web app that runs on a PHP server and does not require a database.

Cloud Notepad on the Desktop

I run a copy on my phone and a copy on my laptop.

The app was built by Kaspars Dambis and I forked it to modify the design and add some usability tweaks like ctrl-S to force sync. You can find the source code of my fork on GitHub if you want to install it yourself.

]]>
/tags/all Mon, 04 Sep 2017 03:00 GMT
Clerk: Event Logging Web App entries/clerk-event-logging-web-app http://mccormick.cx/news/entries/clerk-event-logging-web-app Screenshot

There are times in life when you need to keep track of periodic events as they happen. For example when you have a new baby it is sometimes useful to keep track of when they feed, nappy changes, etc. Another example might be tracking how often you eat chocolate or drink beer.

Clerk is a simple self-hosted web application that I built which you can install to the home screen of your device (by doing "add to home screen" in your browser) or load up on your tablet or laptop. You can then keep track of simple events with two taps on your device - once to open the app and once to record the event.

Screenshot 2

For every event logged the event type, timestamp, and comment are stored in CSV files. Events are stored in individual CSV files - one file per event type. You can also download all CSVs stiched together with an extra column for the event name.

Features

  • Web based.
  • Easy to deploy.
  • Self-hosted & FLOSS.
  • Simple text based CSV format.
  • Allows multiple people to log events.
  • Mobile friendly - "Add to Home Screen" web-app.

Install

To require authentication, first create a password file:

htpasswd -c /path/to/.htpasswd username

Then copy ./example.htaccess to .htaccess and edit it.

Enjoy!

]]>
/tags/all Thu, 06 Jul 2017 02:38 GMT
Lost Town Of Belgium entries/lost-town-of-belgium http://mccormick.cx/news/entries/lost-town-of-belgium

Scout and I made this stop-motion short film.

How we made it:

  • We used an old Android phone with one of those $10 phone tripods to take each frame.
  • We flipped through the stills in Ristretto by hand, recording the output with RecordMyDesktop.
  • We used avconv to convert between formats.
  • We used Audacity to record the audio.
  • We used OpenShot to edit the video.
]]>
/tags/all Thu, 08 Jun 2017 09:06 GMT
A Smart Alternative to `watch make` entries/a-smart-alternative-to-watch-make http://mccormick.cx/news/entries/a-smart-alternative-to-watch-make watch-make is a script that rebuilds your project only when make detects it needs a rebuild, for example when source files change.

Features:

  • Works with any existing Makefile based project.
  • No dependencies apart from make.
  • Passes any arguments to make (such as -C mydir).
  • Silent if there is nothing to build and does not swallow output when there is.

I wrote it in response to this Stack Overflow question.

The source code is hosted on GitHub.

Install it

curl -s https://raw.githubusercontent.com/chr15m/watch-make/master/watch-make > ~/bin/watch-make
chmod 755 ~/bin/watch-make

Full source code

#!/bin/sh

while true;
do
  if ! make -q "$@";
  then
    echo "#-> Starting build: `date`"
    make "$@";
    echo "#-> Build complete."
  fi
  sleep 0.5;
done
]]>
/tags/all Wed, 31 May 2017 03:13 GMT