Standing on giants: the open data behind the field (Foundations, part 8)
Part 8 of Foundations: the sources we compile into one field of language and world knowledge — all of whom deserve the credit — and why licensing was a design decision, not paperwork.
CleverMemory's substrate is a set of knowledge packs: language and world knowledge compiled into one queryable field. I want to be direct about something: none of it sprang from us. It's the accumulated, freely-given work of hundreds of researchers and thousands of volunteers. Credits on the record:
- Wikidata (Vrandečić & Krötzsch, and its volunteer editors) — a hundred-plus million entities and their relations, CC0 public domain. The backbone of world knowledge, right down to metadata describing its own properties.
- PropBank (Palmer et al.) and VerbNet (Kipper Schuler) — verb argument structure and thematic roles. The data behind part 4's events.
- UniMorph (Kirov, McCarthy et al.) — inflectional morphology across hundreds of languages. How "ran" relates to "run."
- Universal Dependencies (Nivre, de Marneffe, and a huge consortium) — consistent grammar annotation for 100+ languages.
- Princeton WordNet (Miller, Fellbaum) and the Open Multilingual Wordnet (Bond et al.) — word senses, multilingually linked.
- Unicode CLDR and the Unicode Character Database — what text itself means. IANA tzdata (Olson, Eggert, and maintainers) — the world's timezone history. Quietly heroic public data.
- QUDT — quantities, units, dimensions, machine-readable. The reason "3 km" and "3,000 m" can be the same fact.
- GeoNames — the planet's places and what contains what.
- GUM (Zeldes and Georgetown students) — richly annotated discourse.
- ATOMIC 2020 (Hwang, Bhagavatula et al., AI2) — everyday commonsense, generously CC-BY.
- Metamath (Megill and contributors) and mathlib (the Lean community) — formal math with every step checkable. Proof discipline, as data.
- Project Gutenberg and the public domain — the corpus commons.
The licensing rule that cost us
Early on we made a rule that hurt: friendly licenses only. Public domain, CC0, CC-BY, permissive. No share-alike, no non-commercial, no research-only — no matter how good the resource. Some famous datasets are missing from that list for exactly this reason. Excluded with regret, and with respect for their authors' right to set their terms.
Why be that strict? Because a knowledge substrate is infrastructure, and infrastructure with murky terms poisons everything built on it. If you download our packs, you should be able to build a product, a paper, or another dataset without hiring a licensing archaeologist. That only works if every row is clean at the root. So every fact carries provenance to its source, and the whole set is auditable — walk any answer's citation back and you land on a friendly origin.
Where the friendly set has gaps, we fill them the honest way: derive from clean sources, generate carefully and check before believing — never by quietly leaning on restricted data. Slower? Yes. Narrower at first? Also yes. Worth it, because here's the whole point: you can't build a common language for language on borrowed words.
That closes Foundations. The two posts after this — on refusing to answer, and on building the packs fast — are where the story turns operational.