Checking Knowledge Graph Extraction with GLiNER2.5-Decide

"The Company does not expect to acquire additional terminal assets in 2026." That sentence has the surface shape of an acquisition and asserts the opposite, which is why it's one of eleven traps planted in the corpus I use for knowledge graph extraction experiments. In the last post, catching sentences like it took a hosted model and an API key. This time it took one forward pass of a local classifier, on a CPU:

modality  denial 0.99   fact 0.00   forecast 0.00   possibility 0.00

That classifier is GLiNER2.5-Decide, which Fastino released this week: the classification model in the GLiNER2.5 family, built on the same gliner2 library as the GLiNER2.5 extractor, with a DeBERTa-v3-large encoder. Given a text and a label set passed at call time, it returns a probability for every label. It finds no spans, generates no tokens, and its model card pitches it for decisions like routing tickets and moderating content. But pick one of these labels for this text is the shape of most of the questions the last post asked TypeSafe's Jev between extraction and the graph. Is this edge asserted? What kind of thing is this mention? Are these two mentions the same company? So I asked them again, locally, with a simple division of labor: GLiNER2.5 proposes, Decide disposes.

The short version: on the stages that check what the extractor already proposed, two local models get F1 within a hundredth of the hosted gate and reach the same perfect entity-resolution score. Asked to find relations, or to decide whether two mentions are the same entity, Decide fails. And the wording that works is its own, not the wording I brought over from Jev.

What you'll learn: how to use a label-set classifier to check a span extractor's output, why label names and even task names are part of the question, how a cheap context-free type can widen entity-resolution candidates without replacing the extractor's type, where a classifier stops being useful, and how to store every judgment in Neo4j. Who this is for: developers building knowledge graph or GraphRAG pipelines who are comfortable with Python. The Jev post is useful background but not required.

The Setup, Briefly#

The experiments repo asks how much of knowledge graph construction you can do with small, local, schema-driven models. GLiNER2.5, a 194M-parameter span model, does joint entity and relation extraction against an ontology you supply at runtime. The corpus here is BUSINESS_NEWS: ten synthetic documents about fictional companies, with 47 gold triples, 13 gold alias groups, and eleven planted modality traps - sentences that look like facts and are not. The ontology has 13 entity types and 15 relations.

GLiNER extracts 214 mentions and 170 candidate edges in 7.2 seconds, the same 214 and 170 the Jev post started from, so every comparison below is like for like. Scored against the gold triples, the raw extraction gets precision 0.279, recall 0.404, F1 0.330. The hosted Jev gate took all three to 0.404. That's the bar.

The Model, In One Call#

A Decide schema is a set of tasks, each a name, a label set, and optionally an instruction. Several tasks over one text are scored in one forward pass. Here's the trap sentence with two heads:

from gliner2.classification import ClassificationSchema
import kgx.decide as kd

judge = kd.DecideJudge()        # fastino/GLiNER2.5-Decide, on CPU

sentence = ("The Company does not expect to acquire additional "
            "terminal assets in 2026.")
schema = (ClassificationSchema()
          .single("modality",
                  ["fact", "possibility", "denial", "forecast"])
          .single("industry",
                  ["logistics", "semiconductors", "energy", "banking"]))

for task, dist in judge.classify(sentence, schema).items():
    ranked = sorted(dist.items(), key=lambda kv: -kv[1])
    print(f"{task:9} " + "   ".join(f"{l} {p:.2f}" for l, p in ranked))

modality  denial 0.99   fact 0.00   forecast 0.00   possibility 0.00
industry  logistics 0.36   banking 0.22   energy 0.22   semiconductors 0.20

The modality head is certain. The industry head leans toward logistics, a fair reading of a sentence that mentions terminals and nothing else.

The model card's entry point, classify_text, keeps only the winner. Asked the modality question alone, it returns denial at 0.979 rather than 0.99, most likely because tasks that share a prompt shift each other's answers, which matters again below. Keeping only the winner is right for routing a ticket and wrong for a pipeline stage, because most of what follows reads the rest of the distribution. So DecideJudge goes through the lower-level Classifier, which keeps a probability for every label.

Three models now appear in this series, with three different shapes:

The Ontology Is the Label Set#

The ontology already is the answer set for each question the pipeline asks. The entity types answer what kind of thing is this?, and for any ordered pair of types, the relations the ontology permits between them answer which relationship is this? kgx.decide compiles both from the same ontology object as the extractor's schema, so the two models share one definition of the graph:

ONTO = kgx.BUSINESS_NEWS

type_s, type_names = kd.type_schema(ONTO)
legal = kd.legal_relations(ONTO)    # (head type, tail type) -> relations
rel_s, rel_names = kd.relation_schema(
    "Northwind Logistics", "Cascade Freight Systems",
    legal[("company", "company")])

Printed, the company-to-company question comes out like this:

15 relations -> 43 legal (head type, tail type) combinations

company -> company
  instruction  How is Northwind Logistics related to
               Cascade Freight Systems?
  labels       subsidiary of, acquires, has stake in, supplies,
               partners with, competes with, no relation

Labels are plain words (financial metric, not financial_metric), and no relation is always offered so a question can answer nothing fits. Any string that goes into a label or an instruction passes through kd.clean, because the classifier rejects parentheses and would otherwise fail on a mention like (NYSE: HLCN).

The Label Set Is the Question#

The first test is the eleven modality traps, plus five plain assertions from the same corpus as controls, because a detector that answers not a fact to everything also catches all eleven traps. The natural first move was to carry the Jev post's questions across as literally as the API allows, so I tried three shapes:

from kgx.typesafe import ASSERTION_LEVELS  # plain data: no SDK, no key

criteria = {k: kd.clean(v) for k, v in ASSERTION_LEVELS.items()}

SHAPES = {
    # the Jev post's four statuses, criteria as label descriptions
    "criteria": ClassificationSchema().single("status", criteria),
    # nearly the wording that fixed the Jev post's Noul
    "yes/no": ClassificationSchema().single(
        "factual", ["yes", "no"], instruction=(
            "If a knowledge graph recorded the relationship this "
            "sentence is about as a plain fact, would the graph be "
            "correct? A sentence saying something will NOT happen or "
            "is NOT planned does not make that thing a fact.")),
    # Decide's own idiom: four short concept labels, no instruction
    "short labels": ClassificationSchema().single(
        "modality", ["fact", "possibility", "denial", "forecast"]),
}

The label set is the question. The Jev post's four statuses are good criteria: as the options of Jev's Choice, they caught all eleven traps with the controls intact. Passed to Decide as label descriptions, they catch every trap by calling three of the five controls not-a-fact as well. (A threshold fitted to these sixteen sentences would separate them, but that isn't a method.) The yes/no is worse: its probabilities overlap, with a trap at 0.56 and a control at 0.24, so no threshold separates the traps from the assertions. That's nearly the wording that fixed Jev's Noul in the last post. Four short words and no instruction at all get all eleven traps and all five controls from the argmax.

The model card's examples point the same way: nearly all of them are lists of short label names like refund_request and speak_to_human. The card does support label descriptions for when a name isn't enough, but here they made things worse. Porting another model's prompts across is the natural first move, and here it's the wrong one.

The pipeline's schema adds a fifth label, not stated, explained in the next section. With five labels sharing the mass, P(fact) can run low even when the answer is right ("Priya Raman is Chief Executive Officer of Northwind Logistics" wins at 0.36), so the gate reads the argmax, not a threshold on P(fact).

Checking 170 Extracted Edges#

Every one of GLiNER's 170 candidate edges gets two questions over the same text:

• modality: is the text a fact, a possibility, a denial, a forecast, or not a statement of this at all? This catches the modality traps: the right relation, presented the wrong way.
• relation: of the relations the ontology permits between the two endpoint types, which does the text state, or no relation? This catches extraction errors: the wrong label, or two entities the text doesn't connect. An edge agrees when Decide picks the relation GLiNER decoded.

The text is the edge's window: the sentences it spans, from its first endpoint to its last, not the whole document, because Decide classifies the text it's given as a whole. A window can run several sentences without ever carrying the claim, which is why not stated exists. And the two questions go in separate passes rather than one two-task schema, because tasks that share a prompt change each other's answers; combined, the modality head lost recall on the gold triples. Here is the core of kd.check_edges, condensed:

legal = kd.legal_relations(ONTO)
shared = kd.modality_schema()           # one schema for every edge
texts, schemas = [], []
for graph in graphs:
    for edge in graph.edges:
        h = graph.mention(edge.head)
        t = graph.mention(edge.tail)
        window = kd.edge_window(graph, edge)  # the sentences it spans
        rel_schema, names = kd.relation_schema(
            h.text, t.text, legal.get((h.type, t.type), []))
        texts += [window, window]
        schemas += [shared, rel_schema]

out = judge.probabilities(texts, schemas)     # one batched call

And the gate built on it:

checks = kd.check_edges(judge, graphs, ONTO)
kept = [c for c in checks if c.keep]    # modality is fact, and agrees

Scoring the 340 texts took 33.7 seconds on a CPU. Decide read 123 windows as fact (25 forecast, 12 not stated, 6 possibility, 4 denial) and picked GLiNER's own relation for 130 of the 170 edges:

The two checks fail on different edges. Here are some stated as fact but with the wrong relation, or none, which no modality question could catch:

For the first, supplies is the gold triple. The other two are relations the text doesn't state; Jev scored both below 0.2 on whether the document connects them at all. The gate never adopts Decide's pick: disagreement is enough to drop an edge, and using the pick would turn a check into relation selection, the experiment further down that fails.

One Edge, Four Windows#

The other failure mode, right relation but not stated as fact, isn't all catches. faces_risk(Halcyon, fines) is one: the only sentence behind it is "may face fines if the Commission ultimately finds against it," and Decide reads all three copies GLiNER decoded (at 0.95 to 0.96) as not stated. But operates_in(Halcyon, Germany) shows up four times in one document, once per Halcyon mention GLiNER anchored on, and each copy gets its own window:

Anchored on the one 18-word sentence that states it, the window reads as a fact. Anchored in a long passage about an antitrust inquiry, it reads as something else. Three copies are dropped, the fact is kept, and the canonical edge survives, which is why checks are keyed on mention ids rather than surface strings.

GLiNER's own confidence predicts neither failure: an edge decoded at 0.97 can be the wrong relation, and one decoded at 0.96 can be a hedge. Span-level confidence is about the decoding, not the claim.

Scoring the Gate#

One extraction pass and one checking pass, scored against the 47 gold triples, with the Jev gate for reference:

Each check alone helps a little. Together they close most of the distance to the hosted gate, at no recall cost: every gold triple GLiNER found survives. The checks combine well because they're close to independent: the 12 edges that fail both are about the 11 you'd expect by chance.

How Much Is the Wording?#

In GLiNER2 the task name is part of the question too: it's encoded into the prompt beside the labels, so modality and statement give different answers. kgx.decide uses modality, chosen for meaning before anything was measured. Here are the alternatives, with everything else fixed:

A spread of 0.383 to 0.404 F1, from a word that carries no information the labels don't, is the honest error bar on the gate, not a knob to turn. The best row is the meaningless one-letter name, which ties the hosted gate exactly; picking it would be fitting the gold set. Three names cost one gold triple of recall, so no recall cost holds for the name I chose, not for every name. For every task name, all eleven traps are caught and the gate beats no gate. Naming the claim in an instruction doesn't help either: the window already is the edge.

It Verifies, It Does Not Find#

The Jev post also tried relation selection: enumerate every co-occurring pair of GLiNER's mentions whose types the ontology connects, and ask which relation the text states, or none. At paragraph scope that's 295 pairs, and asking Decide about all of them with kd.select_relations took 12.8 seconds. Decide picked a relation for 240 of the 295 (81%), where Jev picked one for 36% and said none to the rest.

Jev's selection reached F1 0.279 on its own and 0.294 in union with GLiNER. Decide's union does add recall, up to 24 of the 47 gold triples, but with so many wrong edges that F1 falls well below the ungated baseline.

This is the model that reliably confirmed GLiNER's edges a section ago, and the likely difference is the prior. About an edge the extractor proposed, which relation? usually has a right answer among the labels. About an arbitrary pair, the right answer is usually no relation, and a classifier built for routing, where every ticket belongs to some queue, seems reluctant to choose nothing. Either way, Decide verifies; it does not find. Candidate generation stays GLiNER's job. The Jev post found that a judge is only as good as its queue; here, the judge can't be the queue.

Typing Ahead of Blocking#

The repo's hand-rolled kgx.EntityResolver scores B-cubed precision 1.000 and recall 0.897 against the 13 gold alias groups, and the Jev post traced the missing recall to one cause. Blocking is type-scoped, so a security is never compared with a company. GLiNER types HLCN as a security (a ticker symbol, which the ontology says tickers are) and Halcyon Semiconductor Corporation as a company, and both readings are defensible. It also types Torrent Microsystems and Vantage Energy Partners as company in some sentences and security in others. None of those pairs is ever proposed for a merge. The fix there was a second typing pass that lets blocking see an ambiguous mention under more than one type, and typing is squarely Decide's shape.

Type the Mention, Not the Sentence#

The obvious way to ask is the Jev post's way: give the model the mention's context, and name the mention in an instruction. Jev read the whole document; here the context is the mention's sentence. For Decide, that doesn't work:

Decide classifies the text it's given, as a whole. With the sentence in view it types the sentence, which in these filings is nearly always about a company, and the instruction barely steers it. Given only the surface string, it types the mention: across all 214 mentions, agreement with GLiNER's type goes from 72 to 145.

The bare surface has a price: without context, a homograph like Cascade can't be read. So Decide's type never replaces the extractor's; it's offered to blocking as a second opinion. It disagrees with GLiNER on 69 of the 214 mentions, a mix of real corrections (adjusted EBITDA is a financial metric, not a security) and context-free mistakes (SEC as a security at 0.996). On the tickers the distribution is flat: HLCN comes back company at 0.14, a plurality over thirteen labels rather than a verdict. For a second opinion, whose only effect is to add a candidate the resolver can still reject, that's enough.

A Second Opinion, Not a Replacement#

kd.second_opinion keeps every mention under GLiNER's type and adds a clone under Decide's type wherever the two differ. The resolver runs unchanged over the augmented list, and kd.fold_clones folds each clone back onto its original. A clone merges nothing by itself; the resolver's own scoring still has to accept every pair.

opinions = kd.type_mentions(judge, mentions, ONTO)   # bare surfaces
augmented, clones = kd.second_opinion(mentions, opinions)

res_aug = resolver.resolve(augmented)                # resolver unchanged
second = kd.fold_clones(res_aug.mention_to_canon, clones)

B-cubed 1.000 at precision 1.000, with the resolver untouched: the endpoint the Jev post reached with a hosted model reading each mention in its document. Replacing GLiNER's type does nearly as well on the gold groups, but it would write Decide's context-free mistakes into the graph's labels.

The second opinion made six joins. Three are the point: HLCN into Halcyon, and Torrent Microsystems and Vantage Energy Partners each reunited across their two types. One, Automotive with automotive, could go either way. Two fall outside the gold groups, where B-cubed can't see them, and are debatable at best: Dresden the city folded into Dresden fabrication site, and Cascade brand into the company. The wording of the type labels mattered less than I expected: three different label sets all reached 1.000, and better words only meant fewer spurious clones (51 instead of 69).

Pairs That Must Not Merge#

Widening blocking can only add merges, so it needs testing on pairs that must stay apart. The repo's shopping corpus has eleven among its extracted mentions, like Aurora 14 and Aurora 14 Pro, and the resolver already merges three of them on its own. With the second opinion (122 clones over 318 mentions), it merges four. The new one, Halcyon the brand with Halcyon Buds the product, shows the general risk:

'Halcyon'       GLiNER brand     Decide brand 0.36  product 0.12
'Halcyon Buds'  GLiNER product   Decide brand 0.28  product 0.27

6 accepted pairs join them, every one through a clone, e.g.
  'Halcyon Buds' as brand  ~  'Halcyon' as brand   0.93, head block
  'Halcyon' as brand  ~  'Halcyon Buds' as brand   0.95, head block

GLiNER typed Halcyon Buds as a product. Decide, reading the bare string, calls it a brand by 0.28 to 0.27, a coin flip, so a brand clone enters the brand block beside Halcyon. There, the resolver's prefix scoring accepts the pair above 0.9: the rule that correctly unites Northwind with Northwind Logistics Inc. when their contexts agree. Every step is locally reasonable, and the merge is wrong. What would catch it is a stage that asks are these the same thing? So I asked Decide.

Can It Judge a Pair?#

The Jev post adjudicated pairs with one ordered Score each and made zero false merges on these same traps. I tried three shapes with Decide, on the business-news review band (34 labeled pairs, all true merges) and on the shopping pairs with known answers (11 must-split, 6 must-merge):

None of them is a matcher. Comparing names alone accepts most true merges and six of the eleven traps. Reading both contexts refuses nine of the traps, and more than half the true merges along with them. The ordinal shape almost never puts its mass on same.

With both contexts, the three variant pairs the resolver wrongly merges did come back at or near zero, so I tried the narrowest role I could think of: a veto on pairs the resolver had already accepted, below P(same) 0.1. It fails both ways. On business news it vetoes six pairs as plain as Northwind Logistics Inc. and Northwind Logistics; the gold score survives only because other accepted pairs still connect them. On shopping it vetoes the right links, yet all four false merges still stand: clustering is connected components, so the vetoed pairs stay joined through other paths, and no link on the N600X merge was vetoed at all.

Decide can say what a mention is. It cannot say whether two mentions are the same thing, which is a comparison between two texts, not a label for one. Typing goes in front of blocking, the decision to merge stays with the resolver's scoring, and must-not-merge cases need an ontology rule or a model that reads instructions.

Judgments as Graph Data in Neo4j#

As in the Jev post, everything Decide produced is a typed value with a probability, so none of it has to be spent at the point of decision. Build the graph ungated on the second-opinion clustering, carry the checks up to the canonical edges and the typing up to the canonical nodes, and the gate becomes a WHERE clause.

kd.aggregate_checks picks one representative check per canonical edge, ranked by kept, then stated as fact, then P(fact), so an edge stated clearly once and hedged twice is still stated clearly once. (Ranking on the probability alone is the bug the Jev post caught.) That gives 103 entities and 68 canonical edges, each carrying a check: 57 fact, 8 forecast, 2 not stated, 1 denial. And filtering the annotated graph on keep reproduces a graph built from the gated document graphs, 49 triples each, so it's the same gate, not a similar one.

Loading It#

The repo's kgx.neo4j_io.load_graph does the core of this but knows nothing about the Decide layer, so the notebook writes the load out. The layer is ordinary properties. Each relationship carries its evidence plus modality, p_fact, decide_pick, p_relation, agrees and keep, and each node carries Decide's majority type beside the types GLiNER gave its mentions. The Halcyon node's gliner_types of company and security record that HLCN joined it only because blocking compared across types.

Everything is namespaced, because Neo4j Community has one database and an earlier notebook's graph already lives in it: every id is prefixed decide:, and every node carries a __Decide__ label. The load is a constraint and three statements, run with the rows as parameters:

CREATE CONSTRAINT decide_entity_id IF NOT EXISTS
FOR (n:__Decide__) REQUIRE n.canon_id IS UNIQUE;

// nodes: one statement for every ontology label
UNWIND $rows AS row
MERGE (n:$(row.label) {canon_id: row.canon_id})
SET n:$($entity):$($partition), n += row.props;

// relationships, with the Decide judgments as properties
UNWIND $rows AS row
MATCH (h:$($partition) {canon_id: row.head})
MATCH (t:$($partition) {canon_id: row.tail})
MERGE (h)-[r:$(row.type)]->(t)
SET r += row.props;

A third statement, not shown, creates the Document nodes and links each entity to them. Dynamic labels ($(row.label)) let one parameterized statement write nodes under any ontology label, and need Neo4j 5.26 or later. Every write is a MERGE on the namespaced id, so loading twice changes nothing, which the notebook checks: 113 nodes and 238 relationships in 0.07 seconds, identical on a second load. The notebook also writes an un-namespaced Cypher script for an empty database, and one DETACH DELETE on __Decide__ removes the partition.

Querying the Judgments#

The facts both models stand behind - the gate, as a predicate:

MATCH (a:__Decide__)-[r]->(b:__Decide__)
WHERE r.keep
RETURN a.name AS head, type(r) AS rel, b.name AS tail,
       r.p_fact AS p_fact, r.support AS support
ORDER BY r.support DESC, r.p_fact DESC LIMIT 12

Selected rows, in query order:

It's a filter, not an oracle. PARTNERS_WITH the SEC is an extraction error, since the company is subject to the SEC's inquiry, and both checks let it through: the window states a fact, and of the two relations the ontology allows from a company to a regulator, partners with and subject to, Decide picks the wrong one too.

What the corpus raised but didn't assert - everything a gate would have deleted, still in the graph with the sentence behind it:

MATCH (a:__Decide__)-[r]->(b:__Decide__)
WHERE r.modality IS NOT NULL AND r.modality <> 'fact'
RETURN a.name AS head, type(r) AS rel, b.name AS tail,
       r.modality AS modality, round(r.p_fact, 2) AS p_fact,
       r.evidence[0] AS evidence
ORDER BY r.modality, r.confidence DESC

Selected rows, in query order:

Where the two models disagree about the relation - WHERE NOT r.agrees, ordered by r.p_relation, is a review queue. At the top of it is the headline acquisition, which GLiNER also decoded as partners_with and Decide labels acquires, the gold triple. Someone working the queue sees both readings side by side, with the evidence one property away.

What It Cost#

Everything in the notebook ran on the machine it was open on, with no network calls after the first download:

For comparison, the Jev notebook asked 982 questions in 159 requests, with 236,038 input tokens at 174 ms per request. That's not like for like, since a Jev request carries many questions and the 2,952 texts above cover every experiment in the notebook, not just the pipeline. But the trade is clear: Decide swaps a network round trip and a key for CPU time, and checking every edge in this corpus takes about half a minute. Its outputs are deterministic and don't depend on batch size or on which texts share a batch (bit-identical at batch sizes 1, 8 and 32), so there's nothing to cache.

Decide or Jev?#

Side by side, on the same corpus and the same scoring:

They aren't substitutes. On the stages that label one text against a fixed label set (is this window a fact, is this the relation, what type is this string?), Decide lands within a hundredth of the hosted model's F1, or matches it, locally. On the stages that need finding (selection) or comparing two texts (pair adjudication), it doesn't come close, and those are where Jev did clearly better. A pipeline that wants both would split the work by stage rather than by confidence: Decide for the checks and the typing, a model that reads instructions for the pairs. I haven't measured that combination yet.

What I Took Away#

GLiNER2.5 proposes, Decide disposes. Decide is a verifier and a typer, not a generator and not a matcher, and that division of labor isn't optional.

Write the question in the model's idiom. Short concept labels beat good criteria and a careful yes/no. The span has to be the text, because an instruction naming it barely steers the model. And the task name alone moved the gate's F1 by two hundredths. Porting prompts from another model is the natural first move and, here, the wrong one.

Use a cheap opinion to widen candidates, never to replace. A context-free type is wrong too often to overwrite the extractor's, but as a second type for blocking it's exactly enough to get HLCN compared with Halcyon, at a measured price of one false merge.

Judgments belong in the graph. The gate is WHERE r.keep, and what it would have removed (the proposed acquisition, the fines Halcyon may face, the edges where the two models disagree) is still there to query, with the sentence that produced it.

The Jev post's first lesson was that candidate generation and judgment are different jobs. This one splits judgment again: labeling one text is a different job from comparing two, and a classifier on a CPU does the first well enough to put in front of every edge the extractor proposes.

Resources#

• extraction-knowledge-graph-experiments on GitHub - the repo: notebooks 01 through 15, the kgx library, and the synthetic corpora with gold labels
• Notebook 15: propose, then decide and the kgx.decide module behind it
• GLiNER2.5-Decide on Hugging Face - the model card, with examples of every task shape and the fast-decisions benchmark
• GLiNER2 on GitHub - the gliner2 library both models load through, and the GLiNER2 paper
• GLiNER2.5 - the span extractor that proposes every candidate here
• Typed Judgments for Knowledge Graph Extraction with TypeSafe's Jev - the previous post, and the bar this one is measured against

The business-news and shopping corpora are synthetic, written for the repo with gold labels attached. The companies, people and products in them do not exist.