Supernet

No communication systems we have ever built reaches more people than TCP/IP. It provides developers with a magic dedicated wire able to transmit data to any other place given a number. TCP reads a stream of data and then cuts, wraps, transmits, receives, unwraps, and reorders the data at both ends. TCP can be thought of as a train pulling its carts through a tunnel. Data you put in each wagon eventually comes out the other side in the same order - or not at all. But physically - unbeknownst to many - the route and order of each wagon can differ while in transit. It is the IP address on each wagon that ensures they arrive.

It is this design that billions of people and hundreds of billions of devices use every day to send texts or swipe for another video.

Supernets take a different perspective. IP address become a secondary concern. Instead each wagon - including the routing address - gets a hash address. What that adds is the topic of the coming chapters.

First, considers that in many cases writing code that deals with continuous streams of data - instead of individual wagons - is preferable. That choice between stream or message does not matter when you're considering if a supernet is the right tool. The same bit of TCP code that split, unwrap, and reorder messages to form a stream works in a supernet ³ , providing the same abstraction, solving your problems.

Please take a moment to consider - as similar argument will be made multiple times: Adding a hash to the stack does not obstructs or limits how you go about solving problems. Existing system could continue to function as they are right now. Adding the hash enables new ideas, platforms, and designs. Or in other words - as i hope you'll agree at the end: The lack of an explicit hash is an obstruction to how you think about all your digital communication.

For practical purposes we'll stick with TCP/IP and use it to send a toggle linkspace message. In linkspace each message is a 'point'. A points input determine its hash.

from linkspace import *
point = lk_datapoint("toggle")
print(point.data_str)
print(point.hash)

toggle
V85tNI6cNZKcGh5jeYjQXXCssWrlw8zWTRu5q3TcEYc

The entire document up to this sentence fills up ~ 1/10th of the maximum size of a datapoint. A datapoint can hold up to maximum 2¹⁶-512 bytes, around 65.000 bytes. The hash identifying the data fills up this much memory: 00000000000000000000000000000000. The chances of getting the same hash twice by accident is measured in meaningless improbabilities; such as twice guessing the same atom in the observable universe - at the same moment in time since the big bang. The chances of getting the same hash with the same data are 100%.

import sys

# `echo -n toggle | lk datapoint`
point = lk_datapoint("toggle")
pbytes = utils.lk_serialize(point)
sys.stdout.buffer.write(pbytes)
sys.stdout.buffer.flush()

# wraps a Iterable[bytes] - like stdin or other socket - with lk_deserialize
points = utils.lk_deserialize_stream(sys.stdin)

# `| lk format "[hash:str]"`
for point in points:
   print(str(point.hash))

The best known protocol that solves the problem is HTTP/TCP/IP. The hypertext transfer protocol ( HTTP ) wasn't the first solution, but it hit the right balance between powerful, fast, simple, and pretty.

When you open your browser to and use HTTP to visit http://192.168.1.5:8080/garage/above-door, it opens a stream to 192.168.1.5:8080 and sends a header starting with GET /garage/above-door HTTP/1.1. The header field /garage/above-door is the 'path'.

Reading this field in a HTTP stream can be used to toggle a light.

Linkspace does a similar thing with paths like /garage/above-door, but it differs in a fundamental way.

HTTP Headers like GET ... HTTP/1.1, are part of the message defined by a protocol. The HTTP protocol defines the header and they must only contain valid text, and some values are not valid. If a light is added with the name: 💡, /, or \0 ( the 0 value ), then something somewhere is going to break. To get around this limitation in HTTP its up to developers to pick an additional function like encodeURIComponent or a variant of base64 that encode/decode between (non-)valid text.

Hashes are 'random' numbers - not valid text. Instead of having this distinction of (non-)valid text, all linkspace's fields have a known length allowing for any data to be used.

printf "toggle" | lk datapoint | lk format -j "[hash][data]" | xxd

00000000: 57ce 6d34 8e9c 3592 9c1a 1e63 7988 d05d  W.m4..5....cy..]
00000010: 70ac b16a e5c3 ccd6 4d1b b9ab 74dc 1187  p..j....M...t...
00000020: 746f 6767 6c65                           toggle

print( [ byte for byte in point.hash ] ," are 32 bytes ")
print(point.hash, "are 43 letters in base64 encoding " )
print(lka_eval2str("[hash:str]", point=point), "works using the library - but ABE's main usecase is the CLI")

[87, 206, 109, 52, 142, 156, 53, 146, 156, 26, 30, 99, 121, 136, 208, 93, 112, 172, 177, 106, 229, 195, 204, 214, 77, 27, 185, 171, 116, 220, 17, 135]  are 32 bytes 
V85tNI6cNZKcGh5jeYjQXXCssWrlw8zWTRu5q3TcEYc are 43 letters in base64 encoding 
V85tNI6cNZKcGh5jeYjQXXCssWrlw8zWTRu5q3TcEYc works using the library - but ABE's main usecase is the CLI

Linkspace also has the notion of a path. A datapoint only contains hash and data. a 'linkpoint' has additional fields besides the hash and data. One of those fields is the 'path'.

point = lk_linkpoint("", path=["garage","above-door"])

print(point.path_list)
path ="/".join(point.path_str) 
print(path)

Unlike HTTP, path parts know their length and need no special characters. That means they it can be garage, \0, 💡, or /.

It is also possible to do path=["reply",point.hash]. Had paths instead depended on special characters, the using of a point.hash in a path would force a choice of encoding functions, adding overhead and potential incompatibilities.

A second field all linkspoints have is the 'stamp'. Its the current time in microseconds since UNIX epoch. It offers some ordering between points.

There are two types of passwords.

The first type is used to encrypt and save an identity. Such a password protects the identity so that even if it is stolen, no one can decrypt it or impersonate you without the corresponding password.

The second type of password is shared with others, allowing them to prove they know a specific code word to gain access.

In certain scenarios, the second type is appropriate. These passwords are simple, requiring minimal configuration, no accounts, and no account recovery processes. However, many systems take the naive approach of transmitting the password with every message. This is needlessly insecure.

A better approach is to use the password to generate an “ad hoc” key. With this method, both you and the system (“the light” in our example) use the password to derive the same key. This key is then used to sign a message, proving knowledge of the password without directly sharing it.

In Linkspace, this is accomplished with the lki_adhoc(…) function. This function takes a password and a context to generate such a key. The exact context is explained later.

The common approach with passwords, eg HTTP, is to send them with a message. A long time ago you could even just click http://username:password@127.0.0.1:8080/hello/world to do so. This was disabled in HTTP because it doesn't do encryption, and anybody on the network could see username:password.

If you upgrade to HTTPS ( i.e. SSL/HTTP/TCP/IP ) this still works, but 'upgrade' make it sound much more fun than it is. Installing it requires an extensive ritual, and a small donation.

Encrypting streams with HTTPS would add in a bunch of implicit dependencies, even though encryption wasn't even our goal.

It does let us hide the content of our messages, but this should be an orthoganal and standalone concern. Encryption between two end points is so cheap, simple, and common, that a message routinly are encrypted three or four times without noticing. [Wifi, VPN, SSH, HTTPS, Wireless]

(Ab)using encryption as the method of implied authenticity, instead of signatures, is dogma worth reconsidering.

Lets consider the set-up required behind the curtain. Its inner workings are difficult to parse, but the idea behind it is a good one: In the background HTTPS talks to a bunch of computers, this network of computers share messages that get hashed and cryptographically signed, creating links between them, creating a graph of mutually assured credibility that ties a name like https://example.com together with a public key for everybody. For various reasons this key-naming scheme is only used for servers, it costs money, and users shouldn't bother⁵.

At this moment we've encountered the private/public key. These are used to create 'keypoints'. Keypoints are proof that the points: data, public key, and other fields, were combined into a unit on a device holding the private key. The machine controlling the light is set up to only react to these kinds of points. A device without that specific private key is unable to create a point that toggles the light. Next we'll expand to consider what we'll need to support more people controlling the lights.

Whichever you decide to build, linkspace has two more fields to make it practical for anyone and anything: the group and domain.

lp = lk_linkpoint(data="toggle", domain="example-💡".encode(), group=[0] * 32)
# A group is 32 bytes.
print(lp.group)
# A domain is 16 bytes - left padded with \0
print(lp.domain.utf8()) # strip padding - read as utf8
print(str(lp.domain)) # with padding - ascii byte encoding

pre rest68 Required60 ucontentsize66 
pre rest68 Required60 ucontentsize66 
pre rest68 Required60 ucontentsize66 
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
pre rest68 Required60 ucontentsize66 
example-💡
pre rest68 Required60 ucontentsize66 
\0\0\0\0example-\xf0\x9f\x92\xa1

The group indicates the intended set of recipients. If none is provided the public group is used.

The linkspace library does not include build-in networking. It is up to the exchange method you choose to run that ensures groups do not leak data.

The domain separates one app from another. Domains can be thought of as port numbers, in that they indicate what kind of points to expect.

Together the (domain,group,path) determine a point's "space".

lk linkpoint "domain:[#:pub]" |  lk format "[group:str]"

WT0_F-AX2-wuZZ_K6Mm9BFmexjWVAoWDOADpfL1wws8

Supernets are systems where messages link to other messages by their hash. In linkspace this is the `links` field. Each link exists of {tag,ptr}. The ptr - or 'pointer' - is the hash number of a message, and the tag a word or value to indicate its relationship.

datap = lk_datapoint("toggle")
hello_link = lk_linkpoint( "hello world", links=[("a link",datap.hash)] )
hello_signed = lk_keypoint("", key=key, links = [("first",datap),("second", hello_link.hash)])
print(hello_signed.links)

admin = lki_generate()

request = lk_keypoint(path=["lamp","garage","above-door"], key=key)

permission = lk_keypoint(key=admin, path=["permission", request.hash])

incoming_points = [permission,request]
accepting = {}
for point in incoming_points:
    if point.path0 == b"permission" and point.pubkey == admin.pubkey:
        ptr = point.links[0].ptr 
        if ptr not in accepting:
            accepting[ptr] = True

    else if point.path0 == b"lamps":
        if ptr.hash in accepting and accepting[ptr] == True:
            accepting[ptr] = False # don't allow reuse
            path = "/".join(point.path_str[1:])
            print(f"toggle-light {path}")

Assigning a meaning to paths and links is the art of designing code for supernets.

In our example two issues pop up. First, if the program stops it forgets its accepted requests. Secondly, if a permission arrives before a request it doesn't work.

Both can be solvable by saving points. The first by writing a new point after fulfilling the request indicating it has been handled, and the second by reading points we've saved.

Saving points can be as simple as reading/appending to a file with lk_serialize, lk_deserialize.

Eventually though, walking through every message front to back becomes slower than keeping an index. Feel free to pick the database that works for you. The library includes one by default that keeps a log for receive order, and an index by hash and (group,domain,path,pubkey).

To search this database, or more generally request a set of points, the library contains the query struct. Queries let you add predicates, and can quickly select matching points. They are used to read & watch for new matches in the linkspace database, and when requesting data remotely.