The Human in the Loop

In my last post¹ I looked at the notes of my PKM² as a graph, ran a community detection algorithm on that graph and looked at groups of notes that it produced. The question was whether the notes in a group had some “semantic similarity”. By semantic similarity I mean that they revolve around the same topics or ideas. It certainly looked like they did which could seem surprising since the graph algorithm never looked at the actual content of each note. However the links between notes are based on their content. I personally crafted that meaningful connection through links; the graph algorithm surfaced an aggregate of all my individual decisions of which notes are related to each other.

As promised, I will now go to the other extreme. I will now use a different process that looks only at the content of each note and completely ignores the manually crafted links between them.

From notes to numbers

The approach that I took is fairly common: use an “embedding” model and cluster the results. If you are not familiar, an embedding model is a language model that takes a piece of text and returns a vector (that is, a list of numbers). The interesting bit is that two text pieces that revolve around the same idea produce two vectors that are close to each other. So after the embedding you can go and look for clusters of vectors and very likely you’ll find that the are related.

The actual recipe is a fairly standard one without anything particularly clever. Here it is for completeness:

1. Embed: Feed the text of every note into a local embedding model³ to get a vector that captures its “meaning.”
2. Reduce: Use first PCA⁴ and then UMAP⁵ to reduce the dimensionality of the vectors. This makes clustering easier.
3. Cluster: Find the actual clusters with HDBSCAN⁶. Points that don’t clearly belong to a particular cluster are labelled as “noise”.

At this point I already have the note clusters, which is my desired outcome. Similar to last time I’ll attach a label to each cluster by using a chat model⁷. This adds another model of a different kind to the process but it’s not fundamental to it; it’s just a nice summarization convenience.

Just as before, the clusters seem sensible enough. Some of them are precisely what I would expect: Data Analysis & Modeling (151 notes), Agile Project Management (75), Quantum Computing Technology (20), Software Systems Reliability Engineering (57). Some are slightly surprising at first sight like Intentional, focused work practices (28 notes) and Metrics & unintentional consequences (16 notes). But none of them are really unexpected. Also the placement of each cluster seems appropriate, the center region is broadly related to Data and Programming, the top part is the AI-heavy region (including Almato), the bottom-left is Physics and Maths, the right is creativity and human mind, and the low-right is the assorted fantasy, games and neurodivergence.

I can always quibble with some of the details but all of this seems reasonable. The question is, of course, why is this reasonable? Who or what decided if two notes are similar or not?

A community effort

This exercise is the opposite of the last post. The community-detection from before didn’t look at the content of each note, only at the links between them. This embedding approach never looks at links, only at the content. They both are capable of producing sensible clusters based on the meaning of each note but the reasons for it are completely different.

The interesting question for me is: where is the semantic information coming from? In other words, how are both algorithms able to tell if two notes are similar in their content or not?

In the community-detection case from last time the answer was clear. The semantic information was provided by me. Every time I made a link between notes I was associating two notes. Every link was a decision that changed the semantic content of the whole corpus. The algorithm just surfaced that information in a neat way.

This time the notes are, in a way, speaking for themselves. The linking decisions are not part of the equation, rather only the content inside the note counts.⁸ I just said that the clusters on the community-detection algorithm were enabled by my linking decisions so what’s happening here? did we find a way to cluster notes semantically in a way that’s free of human decisions? I don’t think so. The human decisions are still fundamental but now are much harder to see and audit.

With both approaches you get groups of semantically related notes. Both ultimately rely on human decisions but the who and when are completely different. That’s, to me, the interesting part.

In the community-detection case, the meaning was provided directly by me. Every link was a decision stating that two ideas are related. The clusters reflect that accumulation of judgment calls, which is highly personal and includes all my interests, biases, and blind spots. The same notes might be associated very differently by someone else.

In the embedding case, the meaning was provided by the training data of the qwen3-embedding:8b model. Those models are usually trained with a large corpus of text, which is then compressed into the model parameters. Utlimately the meaning in this case come from the enourmous quantity of individial decisions that were made by the undocumented amount of humans that produced the traning corpus.⁹ The model decides if two notes are “similar” or not based on the statistical patterns it acquired from its training data. This in turn is based on how people generally tend to write about certain topic and which words or ideas tend to co-occur. The interests, biases, and blind spots are still there, but they are a reflection of the collection of individuals that contributed to that training corpus.

So both approaches rely on human decisions. The difference is that one is a single idenfifiable person, me, making clear choices by introducing links between notes, and the other is an opaque, and aggregate mass of decisions collected into a black box you can download. This reminds me of the first post in this series¹⁰ with LLMs being black boxes while reasoning engines (which use knowledge graphs) being completely auditable. There are trade-offs for both approaches and, as always, the real skill comes from being able to use the right tool for the right job. The question is not whether a human was involved in the decision process because it all traces back to human decisions. The real question is whether you need to know exactly why a decision was made and by whom.

There is another way

Now I’m ready for the next step in this series. We previously looked at graph communities which derived meaning implicitly by my own links. Here we saw how embeddings derive meaning from an opaque pre-trained model. A knowledge graph takes a different approach. It asks for a lot of up front decision making, it’s slow and deliberate at first but all this formality allows traceable decisions not only for humans but also for computers. In other words, you get automated decisions, like the embeddings models, but they are are completely traceable, like the manual linking.

I’ll try to “lift” some of my notes into a formal knowledge graph to give them this formal structure and see what a computer can automatically derive based on my explicit rules.

Until next time!