SoftspaceAR | Prototype04
SoftspaceAR | Prototype04
See the true shape of your ideas.
Softspace harnesses spatial computing to help thinkers and makers better see, understand, and develop the ideas that form the heart of their creative projects.
This is the fourth in a series of prototypes we’re releasing as a part of our open development process to build the augmented reality version of Softspace.
Help us invent a powerful new kind of tool for thought by testing these prototypes 🧑🏽🔬, following us on Twitter 🐦, and joining the Softspace Discord 👯♀️.
SoftspaceAR Prototype04 is out! You can download the APK here, then install it on your Quest 2 using SideQuest or the Oculus Developer Hub.
This prototype is an answer to the question: Could a spatial interface let you work with the true shape of your ideas?
To see how this question leads to the design of Prototype04, we need to take a step back and understand what this question even means.
In what sense do ideas have a shape, and why does their shape matter?
The Structure of Knowledge
What is the structure of knowledge? In other words: what are the actual things that constitute knowledge, and how are they related to each other?
There are many possible answers to this question. It could be that knowledge is one very long sequence of information; or a scattered field of points that don’t relate to each other at all; or a Borgesian state space of permutational nonsense through which we rifle for nuggets of Truth.
(I’m going to dodge the notoriously divisive question of what constitutes knowledge in the first place, as opposed to other kinds of information like data, or sensory experience. Let’s go with a Stewartian definition of knowledge for now. It’s the thing that PKM folks think about all the time. It’s what knowledge workers are busy working on. It’s the kind of information that tools for thought help you better create, understand, and harness.)
How knowledge is structured matters, because its structure defines what we consider to be legitimate strategies for seeking and creating it.
For example, if knowledge were actually a single long sequence of information (e.g. the revealed word of a deity), then we would seek knowledge within the canon, and create knowledge by expanding on canonical points.
The biblical reference might make this model seem epistemologically unsound… but consider that this is how a lot of writing is produced. Pieces start out as a small list of points, each of which is then expanded into a collection of points, some of which need further elaboration—all while maintaining the overall single-sequence structure.
Another candidate for the structure of knowledge is the tree: information is grouped into topics, which are themselves grouped into topics, going from the specific to the general. This model is the basis for the Dewey Decimal System, which is used to organize books in libraries. It’s also how most people think of species as being organized: within an evolutionary Tree of Life.
If knowledge is structured as a tree, you should seek knowledge by determining the general category that information would be found in, then follow the path of narrowing subtopics until you get to your answer. The work of creating knowledge would be the work of discovering new pieces of information and deciding which sub-sub-sub-branch it belongs to, or filling in categorical “missing links” between something higher up in the tree with something lower down. This is literally how librarians and taxonomists work.
Knowledge is a Network
The problem is that these models prove incomplete under closer scrutiny.
Biblical hermeneutics is the discipline of interpreting and elaborating on the received word, and is a critical bridge between what’s written in the Bible and important practical questions like “should capital punishment be allowed”? These interpretations are core theological knowledge, but they don’t simply slot into the existing sequence of the Bible. Rather, they can draw on information from across the text—or indeed, other authoritative interpretations—to arrive at their conclusions.
The librarian’s dream is every book in its proper place, neatly arranged according to the single totalizing hierarchy of the Dewey Decimal System. But underneath their calm, hardbound surfaces, books are seething with non-hierarchicial irregularities. Footnotes and citations in the text refer to texts that could be on the next shelf down, or in a totally different category two floors up, forming a vast, nonlinear web of meaning.
Even the evolutionary Tree of Life, which seems to hold true for the vast majority of species that reproduce sexually, proves more convoluted than taxonomists would like. Consider the case of the hoatzin, a bird that defies all attempts to paint an orderly picture of the tree of bird species.
These examples hint at a different model.
If knowledge can be said to have a global structure, there’s a lot of evidence pointing to that structure being a network. Technically, it’s a graph in which pieces of information are the nodes and relationships between those pieces are the edges.
Why might we think this?
Footnotes and citations in texts referencing other texts, or even other sections in the same text, form a nonlinear graph structure.
Mind maps almost always take on a graph-like structure, with node-like concepts connected by lines.
A new generation of graph-based note-taking apps (spearheaded by Roam Research) have become very popular, very quickly, in academic and professional communities that work intensively with knowledge.
Anecdotally, a lot of people who were uncommonly good at working with knowledge also seem to intuit its underlying structure as graph-like.
"Creativity is just connecting things… When you ask creative people how they did something, they feel a little guilty because they didn’t really do it, they just saw something. It seemed obvious to them after a while. That’s because they were able to connect experiences they’ve had and synthesize new things.”
“The physical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be ‘voluntarily’ reproduced and combined… this combinatory play seems to be the essential feature in productive thought… the play with the mentioned elements is aimed to be analogous to certain logical connections one is searching for.”
Last, and never least, James Webb Young:
“An idea is nothing more nor less than a new combination of old elements [and] the capacity to bring old elements into new combinations depends largely on the ability to see relationships.”
Graphs Are Hard
With the graph-like structure of knowledge (hopefully) established, let’s get back to the original question:
Could a spatial interface let you work within the true shape of your ideas?
Ideas are knowledge, and the “true shape” of your ideas is the way that the graph-like structure of knowledge is manifested within the ideas you’re working on.
Analogous to the way that language shapes what we can think, or architectural drawing techniques shape what we can build, our tools and methods for working with knowledge shape what we can come to know.
If knowledge really is graph-like, then the most effective strategies for seeking and creating it would be those that treat it as such. Tools that obscure or ignore its underlying structure would limit our ability to do knowledge work.
Unfortunately, graphs are notoriously difficult to represent usably in the kind of bounded, 2D interfaces that computer monitors encourage. Graphs are large, nonlinear, and multi-scalar, three qualities that 2D UIs struggle with.
And if our flat tools are limiting our ability to work with the true shape our ideas, then they’re limiting how well we can understand and develop them.
Spatial Knowledge Graphs
SoftspaceAR Prototype04 proposes a new paradigm for representing knowledge graphs—one that’s free of the limitations of 2D UIs by virtue of its use of spatial computing.
This paradigm is a 3D spatial canvas in which user can create text blocks.
Any terms in the text which are surrounded by double square brackets are interpreted as topics to which that text refers (syntax borrowed from Roam).
Text and topics are our informational nodes, and the references from text to topic are one of two kinds of possible relation.
The other kind of relation that is possible is inclusion: topics can be expanded, and text added to that topic as a way of defining it.
A 3D forced-directed graph simulation running in the background automatically tries to move related items together, while pushing unrelated ones apart. As the user creates, relates, unrelates, and deletes pieces of information, the system constantly updates the overall shape of the knowledge contained within.
Prototype04 shows some of the advantages of working with knowledge graphs spatially:
A spatial interface offers much more room to display the nodes of a graph. In fact, this even allows the graph to display individual text blocks as nodes, instead of the entire pages that many 2D graph views settle for.
The global shape of the graph is generated from the local, node-to-node interactions. This global form is an important part of the meaning of the knowledge graph, because it suggests what the higher-level cluster or syntheses of the details could be.
Object permanence makes it expected and intuitive that items would be includable in multiple contexts. This makes it straightforward to implement true transclusion.
While working on a given node, you can see direct and indirect connections receding into the distance. This creates ambient contextual awareness, so you always know “where” you are within your ideas.
How to use Prototype04
When you launch the prototype for the first time, a pre-constructed knowledge graph loads.
Click on any text block to start editing it.
Press Return at the end of a text block to create a new text block, and press Backspace when typing on an empty text block to delete it.
As you write on a text block, terms enclosed in double square brackets are interpreted as topic references.
Prototype04 will find or create the topic with the matching title (case-sensitive) and link it to the text block where you typed out the term.
As more topics are created, and more text blocks refer to different topics, the system automatically draws related items together and pushes unrelated ones apart—building up the shape of your ideas as you write them out.
Topics are more than just tags for drawing related text blocks together. Topics are epistemic entities that can contain content.
Click on a topic’s title to expand it into a board. Click anywhere on the board to create a new, pre-contained text block.
You can also grab an existing text item and add it to the topic board, or pull text items off and let them float away.
Clicking on the topic title again, or expanding a different topic, will collapse it and allow the contained text items to float away—while remaining connected with a bold connection line.
A text item can be contained in any number of topics—the system permits true transclusion.
Topics are created automatically when referenced for the first time, and deleted automatically if they are no longer referenced and contain no text.
Your work isn’t stuck in the headset! Each topic automatically exports itself as a markdown file, making it easy to bring your work into the rest of your workflow. These .md files are called “Lenses”.
Known Issues
Text blocks and topic titles that are far away become difficult to read—this is a major limitation. I have an idea for a text LODing system that would address this.
Links are currently represented using a simple straight line between the linked items. These get in the way of text, and don’t tell you much when you can’t see the other end. They generally need design work.
Prototype01 took eight weeks start-to-finish. P02 took four weeks; P03 took five weeks; and this prototype took me six weeks. I want to reverse this trend and get the build cycles back toward four weeks or less.
Next Steps
Currently, topic boards use a single column layout, but there’s no reason why each topic’s internal layout system couldn’t be a full ordinospatial layout like the one in Prototype03.
I think this paradigm will really shine with the (re)introduction of images and PDFs.
The only way to create new content right now is by typing out text, but I will port over the very flexible in-app web browser from SoftspaceVR. This allows you to save out full-res images, copy/paste text, snip any part of the browser window into PNG, and create website bookmarks.
Only one topic can be expanded at time right now, but we’ll probably want to let users keep several topics open at once; this raises the question of how to represent content blocks that are transcluded across multiple expanded topics.
These prototypes write to, and read from, the headset’s local storage, which is a real pain to access. Production builds will use a Dropbox backend at first, and expand to other backend options later (IPFS anyone?).
There’s currently a hard distinction between text blocks and topic items, which isn’t necessary. It would be far more elegant and flexible to allow any content block to become a topic container, so that ideas can be defined recursively.
That’s it for now! This post is the longest one yet, coming in at just under 2500 words, so if you made it this far—thanks and congrats 😄.
If you’d like to support this work, please consider following us on Twitter, trying out Prototype04, and joining our Discord to offer your questions, ideas, and criticisms.
Until next time,
Yiliu
