What is the problem?

A word (lexeme) often has several forms, and the right form needs to be selected for a given sentence. For example, in English, the following two sentences use two different forms of the word “book”:

He read a book this month.
He read seven books this month.

English nouns are usually modelled as having two forms, so that is rather simple and small. With syntactic tables we model it with a table with two options, one singular and one plural (example for the noun “goose”).

Now, for lexemes with two forms this leads to a table with two options. That’s not very problematic.

But in other languages, some parts of speech can have many different forms. And I just came to German adjectives. German adjectives have three comparison levels (positive, comparative, superlative), two numbers (singular and plural), three genders (feminine, masculine, and neuter), four cases (nominative, dative, genitive, and accusative), and three inflections (strong, weak, and mixed). That’s 3x2x3x4x3=216 forms. And three predicative forms.

To see how this can look like, here’s the adjective “braun” (brown) with 147 forms and here’s the page on German Wiktionary that shows all forms for the adjective "schön" (beautiful). Yeah, German Wiktionary moved it to its own page because it would have been so unwieldy on the page of the word itself. I tried to rebuild a German adjective on Wikifunctions, here’s “hoch” (high). It “only” has 129 options, as some of the plural forms collapse along the genders and even the cases sometimes, and thus we can save on these if the functions were careful. But trying to use that in functions lead to trouble with the system (T431734).

And that’s just the first kind of lexeme language and part of speech that happened to be a bit bigger.

One question is, can Wikifunctions scale to such big Syntactic tables with hundreds of options? Or do we need another solution?

In the discussion about the Syntactic type proposal, which was the predecessor to the Syntactic table proposal, the community did raise the point that the current proposal will lead to a lot of unnecessary computation. This argument turned out to be correct in practice. This proposal tries to remedy that issue.

Possible approaches

A. Large tables

We have built a full table with 129 forms for the German adjective “hoch”. This worked, but using them in functions did not (T431734). For now it is unclear whether that is a substantial problem or just a minor bug. And even 129 forms is not the end of it: this Turkish verb on the English Wiktionary has more than 300 forms, and the same one on the Turkish Wiktionary has even more, over 400 forms. In reality, these Lexemes have thousands of forms, and the tables on Wiktionary are woefully incomplete. Lists with so many elements will likely run into recursion limits, as well as being very hard for most users to understand let alone correct.

Formatting tables: formatting content works by formatting every option, as they are all HTML fragments. (This is how the system works right now.)

Compatibility to Syntactic tables: no issue, since they are still just tables.

Scalability: we need to always evaluate every single option, no matter if we need it or not. If a table has a few hundred options, and we bold these, we need to bold a few hundred HTML fragments, and yet in the end we throw all but one out, wasting resources and slowing things down.

Plausibility of implementation: can be done without changes to the backend, but won’t scale in the current system. Making this scale would be quite an effort.

Replacing materialized options with function calls

Now, one general rule in linguistics (which, as is often true, has plenty of exceptions) is that the more forms a kind of word has, the more regular they are. That’s why the forms in the German Wiktionary are all created from three or four inputs, not from hundreds. Same for the Turkish verbs mentioned above, they even have a single input only.

So instead of having to materialize all the options beforehand, i.e. making the whole inflection table available with all the individual options, we could collapse some or all of the table options into functions. Here, to visualize it with a small, artificial example in a made-up language:

Gender \ Number	Singular	Plural
Masculine	blond	blonds
Feminine	blonde	blondes

Any such regular table can always be replaced by a function form from stem(“blond”, gender, number)

One example can be seen with this function implementation for German adjectives, which takes four base forms (the stems of the positive, comparative, and superlative as well as the positive predicative) and creates all the other functions through function calls.

One interesting question becomes, how do we use these function calls in Wikifunctions? If we evaluate them all beforehand, we create tons of computation that is never used (in fact, of the hundreds of options we know that only one will be used!). So we have two questions: how do we represent these function calls, and how do we integrate them in an NLG system that allows us to merge the individual tables / lexemes into larger sentence fragments and eventually sentences?

B. Encode them as text in the fragments

Instead of having all options materialized, we materialize some, if any of the dimensions of the table, and we string encode the function calls for the remaining dimensions in the fragment. So instead of having the four options for the four forms for the made up language above, we have one function, e.g. Z7133, and placeholders for the two arguments, the number and the gender. As the tables get merged, and we figure out with which specific gender and number the table has to agree with, the placeholders for the arguments get replaced with the respective values, until the arguments are all given and then the function can be evaluated and the correct form replaces the function call.

So the single option for the table above could look like this as a fragment:

※λZ7133blondQQλ※

Say, we now learn that the form has to agree with a feminine noun. Now we replace one of the placeholder arguments and it becomes:

※λZ7133blondQ1775415Qλ※

Once we know that the form has to be singular, the second Q also gets replaced with the respective value. Now we can evaluate the represented function call and just replace it with “blonde”.

The exact syntax of how to store the embedded function calls as strings has plenty of room for bikeshedding. Note that, rather intentionally, in order to keep things reasonably simple, we only have a single level of function calls and do not embed these. Also the argument types are strictly limited to a single string followed by Wikidata item references. Both of these restrictions can be lifted with a smarter syntax.

(B2.) One alternative would be to reuse the syntax of embedded functions we already have instead of introducing a new one. So it would look like this:

{{#function:Z7133|blond|Q1775415|Q}}

This would have the advantage of having a more powerful syntax (we have a larger selection of arguments and combinations of arguments). It would have the disadvantage of bringing a syntax designed for wikitext into HTML fragments, which might be confusing expectations, users, and implementation.

Formatting tables: since these are HTML fragments, they can be formatted the same way as the current HTML fragments.

Compatibility with Syntactic tables: merge functions need to be aware of this shift. They need to know when to augment the argument list instead of using options, and when to replace the encoded function call with the actual text. But they are compatible otherwise.

Plausibility of implementation: can be done with the system as is.

C. A new Type for quoted and partial function calls

Instead of using text-encoded function calls in unsuspecting HTML fragments, we introduce a new type for essentially the same behaviour, so a real type representing a quoted, potentially partial function call.

We would have a type that looks a lot like a Z7, but is a different type (e.g. a Z77). This type is basically a closure, if I understand it correctly. Unlike a Z7 it does not get evaluated. It also can have missing arguments. Some core functions for this new type are:

Does the function have all arguments set?
Set a given argument to a given value
Evaluate the function call

These functions would be needed in Option B as well, but there they would work on the string syntax.

A list of quoted, partial function calls would then be part of every Syntactic Option, on a new key. The HTML fragments themselves would have placeholders where the results of the function calls would be inserted.

Note that a quoted and partial function call will be a useful type also outside of this particular use case.

Formatting tables: Since we retain the HTML fragments, we can continue to use the current approach for formatting. Boldening a table means we will still create the HTML fragment that bolds the placeholder, and once the function calls are resolved, the placeholder is replaced with the text.

Compatibility with Syntactic tables: We need to add one key to each option, holding a list of quoted and partial function calls. The merge functions need to know about these, add the appropriate arguments, resolve function calls at the right time, and replace placeholders with the result.

Plausibility of implementation: I think that the backend would require some small changes, but the frontend would need quite a bit of fiddly work.

D. A new type enveloping a lispy representation of function calls

This and the next approach assume we are composing the whole text fragment as a function call. This avoids having any placeholders in the HTML fragment. Once the function call is fully specified, it can be evaluated and the result becomes the HTML fragment.

Here we propose to introduce a very simple new type: anonymous functions (also known as lambdas).

It is quite similar to a Z8, but we drop the identity key, we drop the tests, and we allow for only one single composition as the implementation, represented by an untyped list as described below. So the structure is:

K1: typed list of anonymous argument declarations
K2: type of the return value
K3: untyped list

An anonymous argument declaration also has three keys:

K1: string for the identifier
K2: type of the argument
K3: Wikidata item reference for the meaning of the argument

A very simple representation for anonymous functions is to have an untyped list of objects, where the first object is the function to be called and the following are the argument values. (Basically, LISP syntax. This is also inspired by Mahir’s proposal for semantic units.) if an argument is a function call itself, it is represented by a new list.

Values that are not given yet are represented by a Z18/argument reference.

A list that does not start with a function but has a different type as the first value instead evaluates to itself.

Note that anonymous functions are a useful type outside of this particular use case too. They also cover in their entirety all the use cases for quoted and partial function calls suggested in approach C and are strictly more powerful than these.

Formatting tables:Formatting happens by wrapping the existing function call with the new call, e.g. in order to make a word bold, we take the function creating the word,

[ Z711, “blond”, Q1775415, K1 ]

and turn it into

[ Z36961, [ Z711, “blond”, Q1775415, K1 ] ]

Compatibility with Syntactic tables: there are at least three possible ways on how to use this representation:

D1. We keep the existing structure, and extend syntactic options with a new key that takes a list of anonymous functions. Both the existing key for the list of HTML fragments and this new key may have empty lists. Merge functions would need to know how to use the fragments and the functions correctly. In the end, it should always boil down to just having options with HTML fragments.

D2. On Syntactic options, we replace the existing key with a list of HTML fragments with a list of anonymous functions. Note that a list with HTML fragments evaluates to itself, that means that existing HTML fragments just continue the same way (but would need to be wrapped with the new type).

D3. We entirely replace Syntactic options with an anonymous function call (and drop Syntactic options altogether). All features of options become arguments for the anonymous function call. In that case it would probably also make sense to rename the Syntactic table type, since it's not really a table anymore.

Plausibility of implementation: can be done with the system as is, I think.

E. A new Type for anonymous functions

This approach is basically the same as approach D, but has a more useful internal representation for anonymous function calls. Instead of an untyped list, we use a composition.

The argument declarations are the unbound arguments. We can either make a call to an anonymous function and provide the missing arguments (application), or we can replace one of the arguments with a value, leading to a new anonymous function that has one less argument (currying). Once the anonymous function has no arguments left, it can be evaluated any time.

Formatting tables: we envelop the implementation with the formatting function.

Compatibility with Syntactic tables: E1, E2 or E3, as with D1, D2, D3 respectively.

Plausibility of implementation: the backend should be able to deal with this pretty much as is, if I understand it correctly. The frontend would need some fiddly work, but that should be roughly comparable to approach C.

Discussion

	A	B	C	D	E
Can be implemented right away	++	++	-	++	-
Decent UX with a bit of effort	+	- -	+	-	+
Speed and scalability?	-	+	+	+	+
Allows for composition	+	- -	- -	++	++
How does it fit with the existing table type?	++	++	+	-	-
Does it allow formatting content?	++	++	++	++	++

Denny’s recommendation: while writing up the problem, although I initially wasn’t clear about how to proceed, I am now in favor of eventually moving to approach E but starting with approach D. Semantically, the lispy representation from D can be transformed automatically into an E, once the system supports anonymous functions in its UX, and until then, we can use the lispy representation. The lispy representation is not great from an UX perspective, but the goal is to replace it with something better. But we wouldn’t be blocked on that better UX in order to proceed. Out of inertia, I would prefer D1, but maybe D3 is the better solution as it is simpler.

Honestly, I find it difficult to understand the difference between D and E. However, I agree that these two options might be the best one. Looking at 1/2/3 I would have considered 2 the best because we would not have to worry about merge functions and we could also keep the syntactic tables. Just my two cents, --Ameisenigel (talk) 18:35, 26 July 2026 (UTC)reply

Thank you! @Ameisenigel -- the differences between D and E are:

D can be done right away, with the current system, and for E we would need to implement the appropriate type first in the backend
E would have a much nicer and more usable User Interface than D, and it is expected that E would have much fewer errors than D

But again, I also think that we can migrate from D to E later and kick it off with D for now. Thank you for your thoughts! --DVrandecic (WMF) (talk) 12:18, 29 July 2026 (UTC)reply

Santhosh's feedback

As a native speaker a language with productive morphology with high inflection and agglutination — Malayalam, a word paradigm is a function. A Malayalam noun takes plural marking, then case, then postpositional and clitic material (-ഉം, -ഓ, -ഏ, -ആണ്...), and these stack productively. A verb combines tense/aspect/mood, causatives, negation-as-morphology, adverbial and conditional participles, and then cliticizes further. There is no finite table to materialize. So the proposal's direction — from enumerated options toward function-valued forms - it is the right direction.

In that sense, Proposal A, B and C does not help. D and E are where the proposal converges on the theoretically correct answer, which is that Wikifunctions needs first-class lambdas with partial application. This is, I'd argue, inevitable. Wikifunctions is a functional programming system, and any functional system that grows real use cases eventually discovers it needs closures and currying. E is simply lambda calculus done properly; D is E encoded as S-expressions. The lispy list is homoiconic and trivially manipulable (formatting-by-wrapping, [Z36961, [Z711, "blond", Q1775415, K1]], is genuinely elegant), but it's untyped, so you lose static checking of arity and argument types until evaluation time. This is important when a volunteer contributor is least equipped to debug a failure - at least when our debugging and authoring system is in early days.

One governance issue the proposal may have: Wikidata models lexeme forms as enumerated statements. Once forms are generated by Wikifunctions rather than stored, the source of truth for "what are the forms of this word" migrates partly from Wikidata to Wikifunctions, and the two can disagree. For Turkish and Malayalam this is unavoidable. Wikidata cannot enumerate the forms, but the projects should decide deliberately which system is authoritative for which layer.

While I would chose E as the best of these options, what strikes me most is that the proposal quietly reinvents smart paradigm (Détrez and Ranta, 2012) of Grammatical Framework. Am I right?

While A is not useful as such, it can be helpful in a mixed strategy where functions as source of truth, memoized materialization of frequent forms as a cache layer.

I don't fully get the idea of why we need to start with D and move towards E. By wiki standards(You mentioned it as "...due to inertia.."), D will always remain(nothing temporary in wiki world 🙂️ ). Regarding D1 and D2, I think the html fragment is misplaced. Isn't it a conflation of linguistic realization with presentation markup? Bold-ness is not a property of a wordform; it's a property of the rendered document.

For Malayalam, under materialized tables, lexicographic effort scales with lexemes × forms — hopeless for Malayalam, where Wikidata lexeme coverage is thin and nobody will hand-enter thousands of forms per verb. Under D/E, effort scales with lexemes × principal parts plus a one-time investment in grammar functions. Store a stem and an inflection class; generate the rest. That's the only model under which Malayalam coverage is achievable by a small volunteer community.

But there is a catch, someone has to encode the grammar as functions. This is where I am struggling since 2016. The algorithmic understanding of Malayalam morphology is itself incomplete. It is not because some technical issue my efforts to make the grammar implementation is not able to parse a general corpus. I arrived at about 60% at 2020 and not able to expand further. This is because we are dealing with a living language. Prescriptive grammar takes you only to this far. My attempts to explain the grammar in some computable fashion hit a wall. Scholars are not able to explain the rules behind some patterns. MT systems and LLMs are doing better even though they produce Malayalam that no Malayalam speaker would write. --Santhosh.thottingal (talk) 05:16, 28 July 2026 (UTC)reply

@Santhosh.thottingal - Thank you so much for your thoughts! I agree with much of what you say, and it just strengthens my assumption that this is the right way forward. I agree that E would be better than D, but E requires changes on the backend, and I wouldn't want us to wait this long.

Bold-ness is not a property of a wordform; it's a property of the rendered document.

Agreed, but it comes into play as we assemble the wordforms towards becoming sentences and documents.

the proposal quietly reinvents smart paradigm (Détrez and Ranta, 2012) of Grammatical Framework. Am I right?

Sorry, it wasn't meant to quietly reinvent it, but to blatantly borrow the idea.

The algorithmic understanding of Malayalam morphology is itself incomplete. It is not because some technical issue my efforts to make the grammar implementation is not able to parse a general corpus. I arrived at about 60% at 2020 and not able to expand further. This is because we are dealing with a living language. Prescriptive grammar takes you only to this far.

Malayalam is not the first language I hear that about. Whereas we don't have yet a full prescriptive grammar and morphology of the language, I hope that we can capture more and more sentences with the functions we manage to build, slowly moving forward. The goal is not to capture the whole breath of language, but to specifically be able to create some texts for an encyclopedia. A single genre, only generation, not parsing. And with the functions it would always be able to encode exceptions. So it is a bounded problem.

The question is then how much work is still left.

Again, thank you for your thoughts! I really enjoyed reading your arguments, and it help me sharpen my own understanding. --DVrandecic (WMF) (talk) 12:28, 29 July 2026 (UTC)reply

(Add your comments here)