A Knowledge Graph for Experimental Insights: Connecting Experiments, Themes, and Literature in Neo4j

TL;DR — After 20+ experiments across 10 projects, no human can hold the full picture in their head. We built a Neo4j knowledge graph that ingests experiments, findings, and artifacts from org-mode control files and connects them to research themes and the literature graph. The headline: for our T5-adaptive-sensing theme, 7 findings support it and 8 contradict it across 5 projects — and the polarity matches the actual research state. The graph is a derived view: the org files remain the source of truth, and a drift report keeps the two in sync.

The Problem: Scattered Insights

We run a research portfolio that, at the time of writing, contains 23 experiments producing 42 findings across 10 projects. The insights live in three places: plain-text project files (the control plane), per-experiment JSON artifacts on disk, and paper-reading notes that feed our literature graph. Each file is internally consistent. None of them talk to each other.

The practical consequence is a question we kept being unable to answer quickly: what does our portfolio actually know about adaptive sensing? To answer it, you had to open every project file, scan every experiment heading, read every finding note, and mentally tally which results support the hypothesis and which contradict it. After 20 experiments, that is not a query — it is a literature review of your own work, every time.

This is the same problem that motivates knowledge graphs in the first place: the information exists, it is structured, but it is fragmented across stores. A derived view facilitates insights — by project, by theme, by method, by literature claim.

The Idea: A Knowledge Graph for Experiments

We extended our existing Neo4j literature graph with an experiment layer. The literature graph already held roughly 4,000 production nodes — papers, claims, concepts, methods — and the experiment graph adds to it, rather than sitting beside it. One database, label-separated, sharing the same Method, Concept, and Claim nodes. That sharing is the whole point: it is what allows a single query go from “an experiment we ran” to “a paper that uses the same method”.

The node types are the obvious ones:

• Experiment — a single computational or analytical run, identified by a stable experiment_id, with a project, a status, and an artifact path.
• Finding — a synthesized result from one or more experiments, with an upstream pointer and a verdict.
• Artifact — a concrete output file (a results JSON, a figure, a memo) produced by an experiment or a delegation.
• Theme — a portfolio-level research theme that spans projects and the literature, with its own lifecycle (active, paused, retired).
• Method, Concept, Claim — shared with the literature graph. Resolved by name, so an experiment and a paper that both use the same method land on the same Method node.

The edges connect them:

• Experiment INVESTIGATES Theme
• Finding SUPPORTED_BY Theme and Finding CONTRADICTED_BY Theme
• Experiment USES_METHOD Method and Paper USES_METHOD Method — the bridge
• Experiment PRODUCES Artifact, Experiment SYNTHESIZED_AS Finding
• Project PURSUES Theme

The key decision was to extend the existing shared-core relations to admit the new endpoint types rather than introduce parallel edge names. A finding that supports a theme uses the same SUPPORTED_BY relation that a literature claim already uses against a paper. This keeps the canonical vocabulary¹ small and makes the cross-graph joins natural rather than bolted on.

Themes as the Portfolio-Level Organizing Principle

The most useful construct in this graph is the Theme. Each theme lives as a first-class project in our project management layer, with an ID, a status, a description, and a “pursued by” list of project slugs.

Themes are deliberately not a subtype of Concept. A Concept is a literature object — something a paper is about. A theme is a portfolio object — something our research program cares about and is actively investigating. Conflating them would muddy the Concept namespace and lose the lifecycle semantics (active, paused, retired) that themes need and concepts do not have.

The mechanism that makes themes informative is tagging at the source. Every experiment and finding heading in our org project files carries a :THEME: property listing one or more theme ids. A finding that contradicts a theme carries an additional :THEME_VERDICT: contradict property. The extractor reads those properties and emits the corresponding SUPPORTED_BY or CONTRADICTED_BY edge. There is no NLP step, no inference from prose — the edges come from explicit human-authored tags, which is both the strength (precision) and the limitation (coverage depends on tagging discipline).

The Extraction Pipeline

The graph is populated through the same proposal pipeline we already use for literature extraction.

org/projects/*.org  +  experiments/*.json  +  org/themes.org
        |
        v
extract-experiment-graph          (read-only, stdlib-only)
        |
        v
dedup-merge-proposal
        |
        v
validate-graph-proposal           (canonical vocabulary + evidence gates)
        |
        v
human / OpenCode review
        |
        v
stage-proposal                    (stamps provenance, batch id)
        |
        v
promote-batch                     (human-gated, named, reversible)

The extractor is read-only, deterministic, and conservative: it prefers explicit org-properties and IDs over plain prose and does not invent a node that is not backed by a stable heading, property, or artifact path. Every node it emits carries two provenance fields:

• source_hash — a SHA-256 over the canonical derivation inputs (org heading text plus artifact file content plus artifact mtime).
• derived_from — a list of repo-relative paths the node was derived from.

The source_hash is what makes drift detection possible. Because the hash is deterministic over the inputs, you can re-run the extractor and compare the fresh hash against the hash stored on the node already in Neo4j. If they differ, the source has changed and the node needs re-staging. If the node has no corresponding org heading anymore, it is orphaned. The drift report has three buckets: to ingest (in org, not in graph), to update (in both, hash differs), orphaned (in graph, not in org). Orphaned nodes are flagged for human review, never auto-deleted — a cancelled project may still be worth keeping as history.

Re-staging is idempotent because the importer uses MERGE semantics on the identity keys (experiment_id, finding_id, theme_id, etc.). You can run the same batch twice and the node and edge counts do not change. Combined with the source_hash drift check, this means the graph can be kept in sync with a periodic re-extract-and-compare cycle rather than a brittle event-driven pipeline.

One subtlety: when an experiment re-runs and a finding’s evidence changes, we do not update the finding in place. A new Finding node is created with a SUPERSEDES edge pointing to the old one. The old edges stay attached to the old finding; the new finding gets fresh edges from the re-run. A query for “current findings on theme T” filters to findings with no incoming SUPERSEDES edge. Rollback is trivial — drop the new node, the old one is still there. This costs some graph size in exchange for a clean audit trail, and at the current graph-complexity-level, that is the right trade.

What the Graph Can Answer

Once the pipeline has run across the portfolio, the queries that used to require a manual literature review of our own work become single Cypher hops.

Theme polarity. The headline query, for T5-adaptive-sensing: 7 findings support the theme and 8 contradict it, spread across 5 projects. The polarity is not a vague sentiment score — it is a count of SUPPORTED_BY vs CONTRADICTED_BY edges. And it accurately reflects the research state: adaptive sensing wins at high dimensionality with tight measurement budgets (the scaling experiment showed info_gain beating random by 31.6% at D=16), but the trained EBM posterior failed to provide a useful uncertainty signal on simpler toys, and the 1D radar toy was too simple for adaptive acquisition to earn its keep. Seven supports, eight contradictions, and the contradictions are real findings from real experiments, not noise.

Literature–experiment bridge. The method-overlap query walks Experiment -[USES_METHOD]-> Method <-[USES_METHOD]- Paper. In production, PINN is used by experiments in both newpinns and mmwave-pinn-endtoend projects, and the same Method node is shared with the literature graph. A paper that proposes a method and an experiment that tests it resolve to the same node, and one can traverse from one to the other without a manual lookup.

Coverage gaps. A complementary set of read-only queries surfaces where the next experiment opportunity is: themes scoped in themes.org with no INVESTIGATES edge from any experiment (clean targets for new work), methods used by two or more literature papers but by no experiment (candidates for a replication test), and findings with no upstream SYNTHESIZED_AS edge (untracked work or claims still waiting for an experiment to validate them). These queries perform no writes and are safe to run on a schedule as part of the weekly housekeeping digest.

Limitations

The graph is only as good as the org-mode tagging. A finding that should carry :THEME: T5-adaptive-sensing but does not will be absent from the polarity count. The extractor is deliberately conservative — it does not infer themes from prose — so coverage depends on tagging discipline. We audit this through a drift report, but a silent missing tag is still possible.

The Neo4j Community Edition we run lacks the existence constraints that would enforce required properties at the database level. The unique constraints and indexes work; the _not_null constraints do not. We close the gap at the validator (proposals are rejected if required properties are missing), so there is no data-integrity hole, but it is sub-optimal: the integrity guarantee lives in the pipeline, not in the database.

Most importantly: the graph is a derived view, not the source of truth. If the org files and the graph disagree, the org files are right. The graph exists to make cross-cuts cheap; it does not replace the control plane. This is a deliberate constraint — the moment the graph becomes authoritative, you have two sources of truth and they will drift.