diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 3d0969c6e4..7182b7a54b 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -515,3 +515,35 @@ {"area": "mdl/executor", "date": "2026-09-04", "symptom": "A navigation menu's LOG-OUT item could not be authored and did not survive a round trip. MDL's `menu item` took PAGE or MICROFLOW only, so there was no spelling for it; and ako/TestApp's sign-out menu item read back as a plain `menu item 'Item 5';`, so DESCRIBE -> exec turned a working log-out entry into a dead one \u2014 silently, with `mx check` clean.", "cause": "A menu item's action goes through FOUR places that share no code with the button path: menuActionToGen (menu document, modelsdk), navMenuAction (navigation profile, raw BSON), resolveMenuAction (modelsdk read) and parseNavMenuItem (legacy read). Both writers ended in a NoAction default and both readers left the type name unmapped. Added SIGN_OUT to navMenuItemDef in the grammar (it consumes no qualifiedName, so it is read separately from the PAGE/MICROFLOW switch or an ICON after it is mis-assigned), carried it as ActionType \"SignOutAction\" / NavMenuItemSpec.SignOut, and wired all four. Studio Pro stores the same Forms$SignOutClientAction a button carries: DisabledDuringExecution true, nothing else.", "file": "`mdl/grammar/MDLParser.g4` (navMenuItemDef), `mdl/ast/ast_navigation.go`, `mdl/visitor/visitor_navigation.go`, `mdl/executor/cmd_menus.go` + `cmd_navigation.go` (conversion + printMenuMDL + the show summary), `mdl/types/navigation.go`, `mdl/backend/modelsdk/menu_write.go` + `navigation_write.go` + `navigation_read.go`, `sdk/mpr/parser_misc.go`; example `mdl-examples/bug-tests/captrack-10-sign-out-menu-item.mdl`", "insight": "A round trip closes only if the READER produces the exact string the WRITER consumes \u2014 here both readers had a raw-type-name fallback that looked like it preserved information (ActionType became \"Forms$SignOutClientAction\") while breaking the round trip, because DESCRIBE and the writers key on \"SignOutAction\". A fallback that stores the raw name is not the same as handling the case, and it hides the gap better than a NoAction default would. Also: the same logical action reaches storage through four unrelated switches (two writers x two constructs, two readers), so fixing the button path proved nothing about the menu path \u2014 grep for every switch on the action before calling such a fix complete. Controlled by neutralising both readers and re-reading TestApp: `Item 5 -> sign out` goes back to `Item 5`."} {"area": "mdl/executor", "date": "2026-09-04", "symptom": "`ACTIONBUTTON \u2026 (Action: OPEN_LINK 'https://\u2026')` was written by neither engine: modelsdk refused it, legacy fell through to its quiet default and wrote Forms$NoAction, so the button rendered and did nothing with check, exec and mx check all clean.", "cause": "Same missing-case defect as SIGN_OUT, but with two traps a reference settled and reasoning would not. (1) The STORAGE NAME is Forms$OpenLinkClientAction, while the semantic type is LinkClientAction and the executor stamped `Forms$LinkClientAction` \u2014 a wrong $Type that never reached disk only because nothing could write the action. (2) The address is not a string field but a nested Forms$StaticOrDynamicString. Pinned against 31 Studio Pro link buttons (ako/TestApp, FeedbackModule): exactly five keys, LinkType \"Web\" in all 31, and 6 of 31 DYNAMIC (IsDynamic true + AttributeRef + empty Value). MDL authors the static form only, so DESCRIBE flags a dynamic one instead of printing its address as a literal.", "file": "`mdl/backend/modelsdk/widget_write.go` (clientActionToGen + staticAddressToGen), `sdk/mpr/writer_widgets_action.go`, `mdl/executor/cmd_pages_builder_v3.go` ($Type), `mdl/executor/cmd_pages_describe_output.go`, `cmd/mxcli/syntax/features_page.go`; example `mdl-examples/bug-tests/captrack-10-open-link-action.mdl`", "insight": "gen declares a fourth property on Forms$StaticOrDynamicString \u2014 `Attribute` \u2014 that not one of the 31 stored documents carries. Writing it would be the 'never invent a key' failure: a document mxbuild accepts and Studio Pro cannot open. When gen offers more properties than the references show, the references win. Second lesson, about controls: the SIGN_OUT commit used LinkClientAction as its 'still unimplemented' control, and implementing OPEN_LINK silently invalidated it \u2014 the test then failed for a good reason, but a control naming a specific unimplemented feature has a shelf life. Point it at something structurally unwritable instead (ShowHomePageClientAction: no gen type, no metamodel counterpart, no MDL statement that builds one)."} {"area": "mdl/executor", "date": "2026-09-04", "symptom": "A generated domain model opens in Studio Pro as ONE horizontal line of entities, boxes touching, unreadable at any zoom. Reported on a 40-entity model (ako/CapTrackV2, Mendix 11.13).", "cause": "The default position for a CREATE ENTITY with no `@Position` was `model.Point{X: 100 + len(dm.Entities)*150, Y: 100}` \u2014 same y for every entity ever created, x stepping by 150. 40 entities = a 6,950px row; and 150px is narrower than an entity box, so they also overlapped. Replaced with a wrapping grid in the new `mdl/dmlayout` package, and added `mxcli layout` for a real layered layout off the association graph.", "file": "`mdl/executor/cmd_entities.go` (the default), `mdl/dmlayout/dmlayout.go` (new: GridSlot + Plan), `cmd/mxcli/cmd_layout.go` (new command)", "insight": "The default could not have been much better than a grid, and that is the design point: the first entity of a script is placed before the last one exists, so no create-time rule can see the graph. Layout needs the whole model, so it belongs in a separate pass, not as a side effect of authoring \u2014 and because it necessarily overwrites hand-arranged positions it has to be opt-in with a dry run. Two constraints that are easy to miss: an entity stores only Location and NO Size (Studio Pro derives the box when it draws), so spacing must be estimated from name length and attribute count; and a Mendix position is the box's CENTRE, not its top-left, so placement adds half a box. Determinism is load-bearing rather than cosmetic \u2014 an unsorted walk gives a different diagram every run, which rewrites the unit every time and is exactly the churn ADR-0008 exists to prevent (the test catches it on run 0)."} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "A message definition that reaches a child through a REFERENCESET in the forward direction (holder is the FROM entity) builds with two errors: CE6524 \"The domain model has changed and is no longer consistent with the message definition ... The occurrence of 'X' has changed\" on the definition, and CE0295 \"Association 'X' is not allowed\" on any object mapping element bound to it. `mxcli check` and `exec` are both clean. Reported as ako/mxcli-rest FINDINGS #60, measured on RestLab at Mendix 11.13.0 against a 0-error baseline.", "cause": "`resolveAssociationCardinality` in `mdl/executor/cmd_messagedefinitions.go` decided MaxOccurs from the DIRECTION of traversal alone: forward (holder is FROM) -> 1, reverse -> -1. That is right for a Reference and wrong for a ReferenceSet, which is many in BOTH directions. Fixed by branching on `assoc.Type == domainmodel.AssociationTypeReferenceSet` in the forward arm only; the reverse arm and the refusal for a non-connecting association are unchanged.", "file": "`mdl/executor/cmd_messagedefinitions.go` (resolveAssociationCardinality); test `mdl/executor/cmd_messagedefinitions_test.go`; example `mdl-examples/bug-tests/messagedef-referenceset-cardinality.mdl`", "insight": "The wrong rule was derived from a corpus measurement that was exceptionless and still incomplete: all 927 resolvable associations in the demo corpus are `Reference`, 526 storing 1 and 401 storing -1. That split pins the DIRECTION half of the rule beautifully and says nothing at all about the TYPE half, because the corpus contains no counter-example to look at. The lesson is not 'measure more' but 'name the variable the corpus cannot vary' \u2014 a derivation ranged over one input that never changed, and the write-up recorded the confidence rather than the gap. Worth knowing too that this half DOES have a build error behind it where the direction half does not: getting the direction backwards exposes a list as a single object and builds clean, which is why the original comment reasoned that no error existed for either. And CE6524's own advice ('resolve by refreshing the message definition') is actively misleading here \u2014 it says the domain model moved, when the definition was written wrong a second earlier."} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`drop association Mod.A_B` reported success while a message definition still exposed it, naming nothing. mxbuild then reported CE1613 \"The selected association 'Mod.A_B' no longer exists.\" at the Entity message definition, and `describe` went on emitting the dangling member so a describe -> exec round trip carried the break forward (ako/mxcli-rest FINDINGS #60).", "cause": "`execDropAssociation` reconciled entity access rules (the CE1613 case fixed earlier) but knew nothing about message definitions, which hold the association BY QUALIFIED NAME with nothing keeping the two in step. Added a refusal in `mdl/executor/cmd_associations.go` fed by `messageDefinitionsUsingAssociation`, which walks every collection's whole element tree. It is a refusal and not a reconcile because removing the member changes a published contract, which is the author's call.", "file": "`mdl/executor/cmd_associations.go` (the guard), `mdl/executor/cmd_messagedefinitions.go` (messageDefinitionsUsingAssociation / findAssociationMembers); tests in `mdl/executor/cmd_associations_mock_test.go`", "insight": "The useful part was making the refusal actionable rather than correct. 'Remove the member first' would have sent the author straight into a second wall: `drop member` matches the member's ORIGINAL name (for an association, the target entity's) while an `in` path segment matches the EXPOSED one, so the name they wrote in the definition is the one that does not work. mxcli knows both, so it prints the whole statement \u2014 verified by running the printed text verbatim. Two things a single example would have hidden: an association exposed in BOTH directions produces two definitions to clear, and clearing one leaves the other dangling, so the guard enumerates rather than reporting the first hit; and the walk has to descend the whole tree, since the association that breaks is usually nested rather than a child of the root. Scope is worth stating in the error's absence too \u2014 after the guard, dropping the same association still leaves CE1613 on a microflow retrieve and an object mapping element, which are different consumers and deliberately not covered."} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`alter page Mod.NoSuchPage { ... }` passed `mxcli check -p --references` and was then refused by `exec` with \"page not found\" \u2014 check was the weaker gate. Same for `alter snippet`, `alter layout` and `alter entity` (ako/mxcli-rest FINDINGS #60).", "cause": "`validateWithContext` resolved the ALTER's MODULE but never the document itself, so a misspelled module was reported and a misspelled document was not. Added `mdl/executor/validate_alter_target.go`, called once at the top of validateWithContext, resolving AlterPageStmt (PAGE/SNIPPET/LAYOUT \u2014 one statement type, three kinds, set by the visitor from the keyword) and AlterEntityStmt against the project listings, counting documents the script creates earlier. `scriptContext` gained a `layouts` set, which it lacked.", "file": "`mdl/executor/validate_alter_target.go` (new), `mdl/executor/validate.go` (wiring + sc.layouts); tests in `mdl/executor/validate_alter_target_test.go`", "insight": "Worth fixing even though the finding called it harmless, because the harm is not the wrong answer, it is the inverted contract: the skill promises exec refuses exactly what check rejects, and here exec was stricter, so a script passed every pre-flight and then stopped halfway with the statements before the typo already applied. Two guards are what keep a check like this from being worse than the silence it replaces, and neither is obvious: a module the SCRIPT creates has no listing to resolve against yet, and an EMPTY listing means the backend could not answer rather than that the project has no pages \u2014 reporting not-found from either fails scripts that would have worked. Both have controls. The near-name list is the other half: an error that says only 'not found' leaves the author to diff two spellings by eye, which is the actual cost of a typo."} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`alter enumeration Mod.E add value X caption '...'` had no IF NOT EXISTS, so a script that adds an enumeration value was not re-runnable: the second run errored with \"value 'X' already exists\" and `exec` stopped at that statement, leaving every later statement in the file unapplied (ako/mxcli-rest FINDINGS #60).", "cause": "The ADD VALUE / DROP VALUE actions in `alterEnumerationAction` had no idempotency guard, unlike ALTER ENTITY's ADD ATTRIBUTE / ADD INDEX which already used the grammar's `ifNotExists` / `ifExists` rules. Added `ADD VALUE ifNotExists? IDENTIFIER` and `DROP VALUE ifExists? IDENTIFIER`, carried through `AlterEnumerationStmt.IfNotExists`/`IfExists`, and reported as a skip in `execAlterEnumeration`. The bare forms still error, and the ADD error now names the guard.", "file": "`mdl/grammar/domains/MDLDomainModel.g4`, `mdl/ast/ast_enumeration.go`, `mdl/visitor/visitor_enumeration.go`, `mdl/executor/cmd_enumerations.go`; example `mdl-examples/bug-tests/enum-add-value-if-not-exists.mdl`", "insight": "The reportable harm is not the error message, it is that exec HALTS: one already-present value silently truncates the rest of the script, so the damage is in the statements that never ran rather than in the one that failed. Measured by putting a guarded add after an unguarded one and grepping for the second value (absent). A drop-then-add is not a workaround, which is what forces the guard to exist: the drop fails when the value is absent and the add when it is present, so no ordering is re-runnable. One trap while writing the example: `create or modify enumeration` REPLACES the value list rather than merging into it (an enum with three values, re-declared with one, keeps only that one), so a script whose CREATE precedes the ALTER resets the enum each run and the ALTER re-adds instead of skipping \u2014 the first version of the example passed while never once exercising the guard, which is the same class of vacuous test as an integration test that only ever skipped."} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`exec` fails with `no definition for widget com.mendix.widget.web.fileuploader.FileUploader (run 'mxcli widget init -p app.mpr')` and running that command changes nothing. `check` passes. Also affects Events, Google Tag and Markdown viewer.", "cause": "The project has no widget package for it in widgets/, so there is nothing to extract — and the remedy named in the error (widget init) scans exactly that directory.", "file": "mdl/executor/cmd_pages_builder_v3.go", "insight": "Branch the message on whether the package is installed, using the same FindMPK lookup the template loader makes. The important correction is to the PREMISE: these are not Studio-Pro-bundled widgets mxcli should ship definitions for. Measured — a blank 11.13 project ships 33 widgets and none of these four; installing File Uploader (Marketplace module 235351) takes widgets/ from 33 to 34 and the page then builds with NO widget init, because initPluggableEngine refreshes definitions from installed packages itself. A widget whose package is absent is one Studio Pro cannot use either, so there is nothing to ship. Two dead ends first, both of which looked settled: the .mpk files are NOT in Mendix.Modeler.Core.dll (690 embedded zips, zero widgets.mendix.com hits — the hit was a bare ID string), and a .def.json alone is insufficient because getOrGenerateTemplate derives the template from the .mpk in widgets/, so it only moves the error to 'template not found'. Both were chased before anyone asked whether a Studio Pro user could use the widget at all.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "A widget is the only MDL extension point with no in-language DESCRIBE: microflows, nanoflows, Java actions and JavaScript actions all have `DESCRIBE Module.Name`, while a widget needs the CLI `mxcli widget describe`.", "cause": "No DESCRIBE WIDGET statement existed, so `mxcli widget init` generated markdown to fill the gap — and that generated documentation could drift from what the parser accepts, which is what mendixlabs/mxcli#1036 reported.", "file": "mdl/executor/widget_describe.go", "insight": "Move the description builder OUT of cmd/ into the executor and have both the statement and the CLI call it, so the two cannot disagree — cmd already imports executor, so the dependency direction was already right. Two details worth copying: DESCRIBE WIDGET must work with NO project (DescribeFragment was the existing precedent for the exemption in execDescribe), because 'what can I write here?' is asked before anything is open; and the refactor needs a byte-for-byte output comparison against the pre-refactor binary, which is fiddlier than it looks — `git stash -- ` silently no-ops on an untracked file, and a fresh worktree cannot build because generated embed dirs (skillpacks) are absent. Move the files aside and `git checkout HEAD -- ` in place instead.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "The generated widget .md leads with an 'MDL Example' that fails on its own first line — `mismatched input 'tagcontentcontainer' expecting '}'` — because the generator derives keywords from the .mpk while the grammar accepts a hardcoded nine.", "cause": "The example was assembled from the widget definition with no reference to what the parser accepts, so it could promise any syntax the def implied.", "file": "mdl/executor/widget_describe.go", "insight": "An example is only worth emitting if it is PARSE-VERIFIED. Build it, then run it through visitor.Build and refuse to emit it if it fails; choose the head form and each container by the same probe. That makes the example unable to promise syntax that fails, and makes it widen on its own when the grammar gains ground — no second list to keep in sync, which is the whole defect class here. Two traps found while writing it: numbering matters (two child slots both named slot1 PARSE but are invalid on one page — the parser does not check names, and the .md generator had the identical bug), and required properties of type datasource/attribute/action/expression must be omitted-and-named rather than filled with invented values, since a made-up entity name parses fine and then fails at exec.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "DESCRIBE WIDGET's example asked for eleven bindings on Combo box, across options-source modes that are mutually exclusive — overstating what a reader must supply. Separately, the same example emitted NO properties at all when described without a project.", "cause": "Two independent defects. (1) A widget's properties are 'required' only where the editor shows them, and the required list was used raw. (2) The two description sources spell property types differently — a project .mpk gives 'datasource', the embedded template 'DataSource' — and the example's type switch matched only the lowercase spelling.", "file": "mdl/executor/widget_describe.go", "insight": "Prune required bindings by the visibility rules the description already reports, evaluated against the configuration the example itself describes; types.WidgetVisibilityCondition.Hidden is already exported and MDL-WIDGET10 uses it the same way. Be conservative in the direction of over-listing: an indeterminable condition must NOT prune, and a nested (object-list item) rule must never prune the widget's own property. The ceiling is rule EXTRACTION coverage, not the pruning — combobox reports '16 of 32 editor hide-rules recognized', which is why 11 becomes 6 rather than 2, and why attributeEnumeration survives with zero recognized rules. The casing bug is the more general lesson: the two sources' type vocabularies differ, and a test that only checked 'the example parses' passed vacuously against an example that had been emptied — a control asserting the example still ASKS for a visible binding is what caught it.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`mxcli check` PASSES on two widget mistakes that fail at exec: an unknown widget id (`pluggablewidget 'com.acme.NotAWidget' w1`) and a real container keyword on a widget that has no such container (`group` inside HTML Element). Only a typo'd widget KIND is caught, and that is the parser, not the validator.", "cause": "widgetTypeV3 is effectively the widget-kind validator — the grammar's allow-list is what rejects an unknown kind, so the validator never needed an independent notion of one. validateWidgetTreeIn already computes both facts (parentObjectLists[w.Type] for the container, lookupWidgetDef for the kind) but reports neither; the branch routes to validateStaticWidgetUnknownProps, which checks the properties of a presumed static widget instead of questioning the kind.", "file": "mdl/executor/validate_widgets.go", "insight": "Found while settling Open Question 1 of PROPOSAL_def_driven_widget_bodies.md — whether making the widget body def-driven would cost error quality. It would, but the more useful finding is that the hole is ALREADY open for everything that reaches the validator, so closing it is an improvement today and independent of any grammar change. The general lesson: when a grammar's allow-list is doing validation work, removing it needs the semantic check written FIRST, and the cheapest way to discover what the validator really catches is to find an input that already bypasses the parser — here `pluggablewidget ''`, which parses today and lands on exactly the path the generic form would create.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "Two widget mistakes passed `mxcli check` and failed only at `exec`: an explicit widget id resolving to nothing (`pluggablewidget 'com.acme.NotAWidget' w1`), and a real container keyword on a parent with no such container (`group` inside HTML Element).", "cause": "The GRAMMAR was the widget-kind validator — widgetTypeV3 is an allow-list, so an unknown kind could not parse and the validator never grew an independent notion of one. Neither of these is a keyword the parser checks. isUniversalObjectListKeyword actively SUPPRESSED the second case by treating a container keyword as always-an-item wherever it appeared.", "file": "mdl/executor/validate_widget_kind.go", "insight": "THREE guards make these rules safe rather than a false-positive storm, and each was found by a control or a CI target failing. (1) With NO project the registry holds only the nine embedded widgets, so every real project widget looks unknown — MDL-WIDGET25 must stay silent without -p. Measured the hard way: one example file produced 14 violations, and `make check-mdl` broke seven files, because the corpus is checked WITHOUT a project while I had measured only with one. (2) With a project, LoadWidgetRegistry reads only .mxcli/widgets/*.def.json and does NOT refresh from installed .mpk files, so an id whose package IS installed must be treated as real (same FindMPK lookup slice 1's error message uses). (3) A container is never judged against a parent whose definition could not be resolved. Scope guard (1) to the widget-id branch only — the container rule needs a resolvable PARENT, not a project, and a blanket early return kills it. Two process traps: CHECK THE RULE-ID SPACE first (MDL-WIDGET23 was taken by validate_widget_onclick.go, and its own test caught the collision), and a rule that needs a project CANNOT be demonstrated by a .fail.mdl — make check-mdl runs without one, so the file reports 'negative test unexpectedly passed' and makes a working rule look regressed (the Makefile documents this as #891/#892). Keep the repro a plain .mdl and cover the rule with unit tests. Finally, isUniversalObjectListKeyword was a FOURTH incomplete keyword list (7 against the grammar's 9, missing SCALECOLOR/CUSTOMBUTTON/ALLOWEDFILEFORMAT); the replacement derives the set from the registry.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "Two runs of the SAME mxcli binary over mdl-examples/ produced different `mxcli check` output on 11 of 515 scripts. Same warnings, different order.", "cause": "validateStaticWidgetUnknownProps (MDL-WIDGET07) and the WIDGET17/WIDGET18 validators append one violation per property while ranging over w.Properties directly; Go randomises map iteration.", "file": "mdl/executor/validate_widgets.go", "insight": "Sort the keys (sortedPropertyKeys) at the three sites that emit PER KEY. Leave the two loops that do a case-insensitive LOOKUP and break on first hit: they are only order-sensitive when a widget carries two keys differing solely in case, and either answer is correct. Found not by a bug report but by needing check output as a MEASUREMENT instrument for a grammar change - the noise floor was larger than the signal. A validator that emits per map key is nondeterministic output, and the PR checklist's 'map iteration is deterministic' item covers exactly this. CONTROL: revert the sort and the 8-property regression test fails on the first comparison.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "Once a bare identifier was accepted as a widget type, the typo `htmlelemnt frame (tagName: 'div')` reported '0 errors, 1 warning' - a MDL-WIDGET07 warning about `tagName` - while the correct spelling was completely clean. check exited 0.", "cause": "An unresolved generic widget type falls through to validateStaticWidgetUnknownProps, which validates properties against the builtin vocabulary. It complains about the property because it has already assumed the kind is a built-in.", "file": "mdl/executor/validate_widget_kind.go", "insight": "The AST must record WHICH grammar alternative matched (ast.WidgetV3.TypeIsGeneric), set from the parse tree in the visitor - never inferred by comparing the type text against a list of known widget names, which would reintroduce the list the change exists to remove. A generic type that resolves to nothing is MDL-WIDGET25/26 (kind is wrong), and property validation must be SUPPRESSED for it or the message points at the wrong token. Generalises: when a grammar is loosened, the check the parser used to perform must move to the validator in the same change, or a parse error silently becomes a wrong answer. CONTROL: the correct spelling must stay completely clean under the same conditions.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE PAGE on a page mxcli itself authored emitted only the widget head: `htmlelement frame (tagName: 'div', ...)`. The child slot `tagcontentcontainer body { dynamictext t }` and the object list `attribute a1` were absent, silently, at exit 0 \u2014 so describe -> edit -> exec DELETED the widget's body.", "cause": "The write path is correct (the stored BSON carries tagContentContainer with its DynamicText and attributes with the data). DESCRIBE's generic pluggable branch reconstructs only object lists of the chart shape extractObjectLists was built for, and reconstructs child slots not at all. The gap predates the fix but became reachable the moment slices 2-3 made those containers writable from MDL.", "file": "mdl/executor/cmd_pages_describe_omitted.go", "insight": "Until reconstruction exists, emit the gap as an MDL comment rather than a bare head that reads as complete (unreconstructedContainers). Two traps in doing that. (1) getBsonArrayElements STRIPS the leading typed-array marker, so an empty container is length 0 after stripping and length 1 in the raw BSON \u2014 checking the raw length reports every widget as lossy. (2) Warn on CHILD SLOTS HOLDING WIDGETS only, not object lists: a widget template ships DEFAULT entries in its lists that are structurally identical to a user's, so the first version named `event` on a page that never wrote one. A note that fires on defaults is noise and trains people to ignore the notes that matter. CONTROL both ways: 16 real pages in testdata/expr-checker produce 0 notes, and the authored page produces exactly one naming tagcontentcontainer.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE PAGE kept emitting `pluggablewidget '' name` for every widget after being changed to prefer the MDL name, with no error.", "cause": "LoadWidgetRegistry wants the .mpr PATH; LoadUserDefinitions takes filepath.Dir of it internally. Passing filepath.Dir(ctx.MprPath) looked one level above the project and found no definitions, so every lookup missed and the code fell back exactly as designed.", "file": "mdl/executor/exec_context.go", "insight": "A correct fallback hides a wiring bug perfectly: the output stays valid, so nothing fails and no test goes red. When adding a 'prefer X, else Y' path, assert the X branch is actually reached on a real project, not just that the output parses. Related: the registry is keyed BY MDL NAME, so two definitions claiming one name leave Get and GetByWidgetID disagreeing and All() cannot see the collision at all \u2014 guard by round-tripping the name through the same lookup the builder uses, never by counting definitions.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE PAGE dropped an entire object list from a pluggable widget: HTML Element's `attributes` (holding a real value) and `events` were absent from the output, silently, at exit 0. Chart series described fine, so it looked widget-specific.", "cause": "extractObjectListItem tests a value's fields in order and the ACTION branch continued UNCONDITIONALLY once `value[\"Action\"]` merely existed. A widget value carries every field it could have, and Action is always present as a Forms$NoAction \u2014 so that branch consumed all six sub-properties of an item, the item ended with zero Props, and the caller's `len(item.Props) > 0` filter dropped it, taking the whole list with it.", "file": "mdl/executor/cmd_pages_describe_objectlist.go", "insight": "A branch that consumes on KEY EXISTENCE rather than on EXTRACTION is the bug shape; guard on len(map) > 0 and continue only when something was produced. Two measurement lessons cost more than the fix. (1) My first control used python str.replace with no assertion, matched nothing, and was VACUOUS \u2014 it 'passed' while changing no code, the same trap as a test that only skips. Assert the replacement applied. (2) The second control flipped the three conditions but missed that a `continue` had MOVED inside the Action branch, so it reverted the wrong thing and pointed at DataSource; only `git diff -U0` filtered of comments revealed the moved line. Isolating one branch at a time gave the answer: reverting Action alone takes object lists 2 -> 0. A confident wrong root cause in a comment is worse than none.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "A child slot on any pluggable widget other than Gallery was absent from DESCRIBE PAGE, so describe -> exec DELETED the widget's body. Measured: 4 widgets before the round trip, 1 after.", "cause": "DESCRIBE reconstructed a Gallery's `content` and `filtersPlaceholder` by asking for those property keys BY NAME (extractGalleryWidgetsByPropertyKey) and had nothing generic. Invisible until slices 2-3 made such a slot writable from MDL.", "file": "mdl/executor/cmd_pages_describe_childslots.go", "insight": "A child slot is ANY property whose Value holds a Widgets array \u2014 read it off the document instead of looking the key up by name, and a widget nobody has thought about round-trips for free. Skip empty ones: getBsonArrayElements strips the typed-array marker, so empty is length 0 here and length 1 in raw BSON, and emitting them puts a `slot { }` on nearly every widget. NOTE the round trip converges rather than being a fixed point on the first pass: the writer emits an item's properties in a different order than the original document, so describe #1 != describe #2 but describe #2 == describe #3. Describe itself is deterministic (5 identical runs). A fixed-point assertion alone would NOT have caught the original bug \u2014 an empty describe is also a fixed point; assert that the CONTENT survives.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`ValueAttribute: Total` on a PieChart/HeatMap was reported MDL-WIDGET01 \"has no property `ValueAttribute`\", and because exec refuses a script with errors the page could not be written at all. The value itself persisted correctly (DESCRIBE returns `seriesValueAttribute: Total`).", "cause": "allowedWidgetProperties built its set from each PropertyMapping's PropertyKey and Source, never its MdlAliases. The def.json declares mdlAliases:[\"ValueAttribute\"], the BUILDER resolves through it (widget_engine.go), and the knownProperties set in widget_defs.go walks it — the validator was the only one of three readers of the same def.json that did not.", "file": "mdl/executor/validate_widgets.go", "insight": "When a def.json field is consumed in more than one place, the checker is the one that gets forgotten, because a checker that is too strict still 'works' until someone writes the documented syntax. Grep for every reader of a field before adding a fourth. The give-away here: the file's own header claimed `mxcli check`-clean, which was true only because `make check-mdl` runs WITHOUT a project, so no widget definition loads and the whole rule is inert.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "MDL-WIDGET10 warned `dynamicDataSource is hidden when its own dataSet is \"static\" — the value will be ignored` on every chart series written the documented way. 11 warnings on 34-chart-widget-examples.mdl, one per series, none of them real.", "cause": "A chart's two datasource sub-properties SHARE the Source name \"DataSource\" (measured on linechart.def.json: staticDataSource and dynamicDataSource both declare it, neither declares an alias). itemValueMap resolved by Source, so the one friendly `DataSource:` marked BOTH explicit. buildObjectListItem routes on dataSet (seriesDataSourceMatchesMode) and writes only one.", "file": "mdl/executor/validate_widget_hidden.go", "insight": "A Source is not a unique key. Where two mappings share one, the checker has to reproduce whatever disambiguates them in the WRITER — here the dataSet mode — or it reports properties the script never wrote. Scope the gate to the case the writer scopes it to (chart series datasources); a general 'skip shared sources' would blind the rule everywhere else. Control that catches over-reach: a non-chart item property sharing a Source must still resolve.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`check -p` on a view entity reported \"attribute 'Units': declared as Integer but OQL expression 'sum(s.Units)' returns Decimal. Fix: change to 'Units: Decimal'\" — and following that hint BREAKS the build.", "cause": "inferAggregateType's SUM branch fell back to Decimal when the argument type could not be resolved. The argument is unresolvable exactly when the source entity is created by the same script, since check skips references to script-created objects — i.e. the common shape for a view entity. inferTypeStatic's own SUM branch already said 'return Unknown, do not guess Decimal'; the project-aware path disagreed with it.", "file": "mdl/executor/oql_type_inference.go", "insight": "Measured on mxbuild 11.6.6, two views over the same sum(s.Units) where Units is Integer: declared Integer = 0 errors, declared Decimal = CE6770. The diagnostic inverted the truth, so it did not merely cry wolf — its Fix: walked a working project into a broken one. When a fallback has to guess, return Unknown; a skipped column is a missed error, a wrong guess is a manufactured one. The control that makes the mxbuild evidence mean anything: a deliberately wrong column type in the same app DOES fail CE6770, so mxbuild was really validating view entities.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "Three charts on one page, each with a series the author named `s`, failed check with \"duplicate widget name 's' (used 3 times) — Mendix requires unique widget names per page (CE0495)\". mxbuild 11.6.6 reports 0 errors on the same page.", "cause": "checkDuplicateWidgetNames counted every named node in the tree. An object-list item (a chart `series`, a gallery `customitem`) is a WidgetV3 child in the AST but a WidgetObject in the model, and the model stores no name for it.", "file": "mdl/executor/validate_page_context.go", "insight": "The proof that a name is not stored is free: author `series sRegion` and DESCRIBE it back — it returns `series series1`, because DESCRIBE has to synthesise what the document does not carry. A name the model does not hold cannot be a CE0495 duplicate; the same reasoning already excluded rows and columns in widgetKindsWithoutStoredNames. Read the container keywords from the parent's def.json objectLists rather than listing them — the containers are def-driven, so a keyword table would drift the moment a widget ships a new list. The rule needs the registry threaded into validatePageContextTree, and must fall back to its old behaviour when there is none (check runs with no project in CI).", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE WIDGET's generated example named properties mxcli's own validator rejected with MDL-WIDGET01 \"has no property\" — 33 across combobox/gallery/barcodescanner/image, 78 over the whole property surface. Since exec refuses a script with errors, barcodescanner had no legal spelling: name them and exec refuses, omit them and mxbuild reports CE0463.", "cause": "Two readers of one widget. DESCRIBE parses the project's .mpk; the validator reads the WidgetDefinition. They agree for most widgets because the .def.json cache is GENERATED from the .mpk — but nine widgets are hand-crafted in sdk/widgets/definitions/ and deliberately never extracted per-project, so their property list is whatever someone typed (combobox: 73 in the .mpk, 7 mapped + 4 known).", "file": "mdl/executor/widget_known_props_from_mpk.go", "insight": "The fix is KNOWN, not ALLOWED: an unmapped .mpk property becomes MDL-WIDGET06 (\"recognized but not yet persisted; a non-default value will be dropped\"), never silently accepted by a write path that does not exist — trading a false error for a silent drop is the worse bug. Apply it to every definition rather than to the nine: recomputing a generated def reproduces what generation put there, so it is idempotent where redundant, and naming the nine is the same hand-maintained list one layer up. The guard to write is the ROUND TRIP between the two readers (every property DESCRIBE emits is one the validator does not call nonexistent), not \"the example parses\" — it already parsed.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "MDL-WIDGET10 warned that a property \"is hidden … the value will be ignored\" on 32 of mxcli's OWN generated widget examples — every warning naming a property the example had just written itself (videoplayer emits heightUnit: 'aspectRatio' then the height that choice hides).", "cause": "Two independent bugs with one shape. (1) hiddenUnder gated only the branch asking for a BINDING and not the scalar branch that writes literals, so half the properties skipped the narrowing. (2) exampleValues resolved values from the .mpk while the validator resolves them from the WidgetDefinition's mapping — a selection property has no defaultValue in the .mpk, so gallery's itemSelection looked indeterminable.", "file": "mdl/executor/widget_describe.go", "insight": "When a generator and a checker implement the same rule, they must read the same FACTS, not just run the same logic — mirror the checker's value resolution exactly, including its implicit fallbacks (a selection with no default is written as None; that is the builder's behaviour, not a guess). Two tests are needed because one is blind: the structural one can only see what the generator itself considers hidden, so the residue from cause 2 is invisible to it — only building the example, parsing it and running the REAL validator catches that. And the control must assert pruning FIRES (29 scalars pruned across 36 widgets): \"emits nothing hidden\" is also satisfied by emitting nothing, and by a hiddenUnder that always returns false.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "MDL-WIDGET08 rejected mxcli's own generated example: \"property `dataSet` has invalid value `…` — valid values are static, dynamic\", 11 of 42 widgets. The example block claims 'parses as written' and did parse; it did not CHECK as written.", "cause": "The object-list item line was built as ItemKeys[0] + \": '…'\" — a hardcoded placeholder — instead of deriving a literal the way the widget's own scalars do.", "file": "mdl/executor/widget_describe.go", "insight": "Two sources had to be consulted and the second is the lesson: propsFromMPK carries item sub-properties as Children with their enums, which covers most widgets, but ParseMPKForWidget returns 0 children for a PopupMenu's basicItems while the DEFINITION carries {\"propertyKey\":\"itemType\",\"value\":\"item\",\"enumValues\":[...]}. The definition wins, because it is what MDL-WIDGET08 checks against. Scope the assertion to what the checker rejects — \"no MDL-WIDGET08\", NOT \"no ellipsis anywhere\": a free-text sub-property has no correct value to invent and the validator accepts any string, so a placeholder there is honest output and testing for the character would be testing the wrong thing.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "CI-only test failure: 'no widget has an authorable object list with item properties — nothing here would exercise the item literal'. Green locally, red on a fresh checkout.", "cause": "The test built its registry from testdata/expr-checker, whose .mxcli/widgets/*.def.json cache is DERIVED and gitignored. Locally that supplies 33 definitions (charts, DataGrid2, HTML Element — the ones with object lists); in CI only the hand-crafted definitions in sdk/widgets/definitions/ load, and none of those declares an authorable object list with item properties.", "file": "mdl/executor/widget_example_item_literal_test.go", "insight": "Reproduce a CI-only failure by moving the gitignored artifact aside (`mv testdata/expr-checker/.mxcli/widgets /tmp/...`) — instant and exact, no pushing to find out. The deeper rule: a fixture's COMMITTED inputs (widgets/*.mpk) are fair game, its derived caches are not, and the split is invisible until CI. Structure the pair as hermetic-test-carries-the-guarantee + end-to-end-SKIPS-when-it-cannot-run, rather than a single test that Fatals on an environment it does not control. Note the failure was the test's own vacuity control firing correctly — that is the control doing its job, not a false alarm, and the fix is to make the assertion runnable everywhere rather than to weaken the control.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`CHANGE $Order (\"IsArchived\" = true)` naming an attribute the entity does not have passed `mxcli check -p --references` (exit 0), passed `exec` (\"Created microflow\"), and surfaced only at the far end of a build as CE1613 \"The selected attribute 'Bench.Order.IsArchived' no longer exists\" (mendixlabs/mxcli#1048).", "cause": "Reference checking resolved the DOCUMENT and the ENTITY a statement names and stopped there; member names inside a create/change were never resolved. exec does resolve them (`resolveAttributeInEntityHierarchy`) but on failure writes `.` anyway, and that fabricated identifier is what mxbuild rejects. Added `mdl/executor/validate_member_refs.go`, called from validateProgram, walking the entity and its generalizations.", "file": "`mdl/executor/validate_member_refs.go` (new), `mdl/executor/validate.go` (wiring); tests `mdl/executor/validate_member_refs_test.go`; example `mdl-examples/bug-tests/member-refs-resolved-at-check.mdl`", "insight": "The design point is that a CHECK needs three outcomes where exec needs two. exec can answer resolved/not-resolved because it has a fallback either way; a check that turns 'I could not look' into 'your attribute is missing' is a false error blocking a script that builds cleanly, so 'could not establish' is a third state and only the middle one is reported. Worth recording what that guard is NOT for: I first justified it with the System module and was wrong \u2014 GetModuleByName/GetDomainModel answer for System perfectly well (System.FileDocument loads with its 6 attributes, so inherited members resolve), and #1047's System failure is a different path that loads the unit by id out of mprcontents. The real motivation is that mxcli has several backends and every lookup may error. THREE separate times in this work a green result turned out to be vacuous, which is the transferable lesson: a corpus sweep reporting 0 false positives across 14 real scripts (the check never fired \u2014 every CHANGE target was bound by an unmodelled activity), an example that passed while exercising nothing (a `skip entities the script creates` guard skipped the whole self-contained file), and the same example silently inert against a virgin project (members are resolved against the model, so there must BE a model). Each was caught only by perturbing a known-good input and demanding the check fire; none would have been caught by adding more assertions."} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "A widget's XPath constraint naming a member the entity does not have passed `check --references`, passed `exec`, and failed the build: `where [Bench.Order_Status = 'Open']` on a datagrid over Bench.Order gave [CE1613] \"The selected association 'Bench.Order_Status' no longer exists.\" (mendixlabs/mxcli#1049).", "cause": "Reference checking resolved the entity in `database from Mod.Entity` and never looked inside the `where [\u2026]`. Added validateXPathMembers in `mdl/executor/validate_widget_member_refs.go`, walking the constraint with the same step-by-step traversal mdl/xpathrefs performs for renames: a bare step must be an attribute of the entity it lands on, a Module.Name step an association or an entity.", "file": "`mdl/executor/validate_widget_member_refs.go` (validateXPathMembers), `mdl/executor/validate.go` (wiring); tests `mdl/executor/validate_widget_member_refs_test.go`", "insight": "The false positive that a control caught: reporting a qualified step requires the BASE entity to be known first. Without that guard, a page whose entity the project does not have (a module the script creates, or the wrong app) reported a perfectly real association as missing. Same three-valued discipline as the create/change member resolver, and it has to be applied at every reporting site rather than once \u2014 the bare-member path inherited it from resolveMemberOnEntity and the qualified path did not, so only one half was safe. Also: the XPath parse runs with ANTLR's error listeners removed and can return a tree that silently omits part of its input, so an unparseable group is skipped rather than reported \u2014 xpathrefs documents that trap for the rename path, where the cost is a corrupted constraint; here it would be an error about text nobody wrote."} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`ContentParams: [{1} = $Customer/Name]` passed check and wrote the whole string into the attribute name: [CE1613] \"The selected attribute 'Bench.Customer.$Customer/Name' no longer exists.\" (mendixlabs/mxcli#1046).", "cause": "A template parameter is evaluated against the widget's own context object, so a variable root has nowhere to resolve. Added MDL-WIDGET24 in the NO-PROJECT pass (ValidateWidgetParamPaths) \u2014 the answer is in the statement, so putting it under --references would leave `mxcli check page.mdl` silent on a mistake it can see.", "file": "`mdl/executor/validate_widget_member_refs.go` (ValidateWidgetParamPaths / templateParamDefect), `mdl/executor/validate_program.go` (wiring)", "insight": "The rule is NOT 'a variable root is wrong', and getting that from reasoning rather than measurement would have shipped a false positive. Five shapes, each executed and built on 11.13.0: `OrderNo` clean; `Assoc/Attr` clean; `$currentObject/Assoc/Attr` CLEAN, because resolveAssociationAttributePath strips that one prefix; `$currentObject/Attr` CE1613, because the strip only happens on the association branch; `$Var/Attr` CE1613 for any other variable. Two further facts fell out: the legal association form is two segments (`Assoc/Attr`), not the XPath three (`Assoc/Entity/Attr`, which also fails); and the check immediately found a SHIPPED bug-test example (ledger-27) that used `$currentObject/Amount`, had only ever been run through `mxcli check` with no project, and built with six CE1613s the first time it was executed \u2014 a validator finding a defect in the corpus meant to validate it."} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`UPDATE SECURITY` was inert on every MPR v2 project: it returned on the first module it could not reconcile, and System always is one \u2014 `Error: failed to reconcile security for module System: load domain model 00000000-...-002: .../mprcontents/00/00/...002.mxunit: no such file or directory`. Reported as \"UPDATE SECURITY does not fix the CE0066 it exists to fix\" (mendixlabs/mxcli#1047). A second defect in the same command: `update security RestLab` (without IN) parsed cleanly, dropped the module name, and ran project-wide.", "cause": "execUpdateSecurity returned mdlerrors.NewBackend on any ReconcileMemberAccesses failure. System's domain model is SYNTHESIZED rather than stored (no unit file behind the id the module carries), so loadDomainModelGen can never read it. Now System is skipped by name \u2014 reconciling it is not merely impossible but wrong, its access rules are the platform's \u2014 and any other unreadable module is reported and stepped over. The grammar's `(IN qualifiedName)?` became `(IN? qualifiedName)?`, so a bare module name scopes instead of reaching ANTLR's error recovery.", "file": "`mdl/executor/cmd_security_write.go` (execUpdateSecurity), `mdl/grammar/domains/MDLSecurity.g4`; tests `mdl/executor/cmd_security_update_test.go`; example `mdl-examples/bug-tests/update-security-runs-at-all.mdl`", "insight": "Two different System failures live in this codebase and conflating them wastes a diagnosis \u2014 I did it once in this same session. GetModuleByName/GetDomainModel answer for System perfectly well (System.FileDocument loads with its 6 attributes), which is why inherited-member resolution works; loadDomainModelGen, which reads the UNIT BY ID out of mprcontents, cannot, because System has no file there. Same module, opposite answers, different call path. The other half is the more interesting bug class: `update security RestLab` was not a parse error but a SILENT SCOPE ESCALATION \u2014 the user asked to touch one module and the command touched all of them, with `mxcli check` reporting Syntax OK. Worth looking for wherever an optional keyword precedes an optional operand. Finally, the report's stated precondition (adding an attribute leaves one CE0066) does NOT reproduce: measured on 11.13.0, `alter entity ... add attribute` on both engines and a whole-entity rewrite each left the project at 0 errors, because every mxcli write path already reconciles. So the command could not be shown repairing a real CE0066 end to end \u2014 the integration example says so explicitly rather than implying coverage it does not have."} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "A datagrid whose datasource is an association path at PAGE level typed its rows as the entity it navigates AWAY from: `datagrid gA (datasource: $Customer/Bench.Order_Customer)` bound its columns against Customer, giving [CE1613] \"The selected attribute 'Bench.Customer.OrderNo' no longer exists.\" at Columns (1/1). The SAME path inside a data view was correct (mendixlabs/mxcli#1045).", "cause": "resolveAssociationDestination picks the end opposite the context, and the context passed was pb.entityContext \u2014 the ENCLOSING data container's entity. At page level nothing encloses the widget, so it is empty, neither end matches, and the function's last-resort fallback (\"default to the child (TO) side\") returns the entity the grid started from. Fixed by resolving from the NAMED context variable when there is one: `$Customer/\u2026` traverses from whatever $Customer holds, enclosed or not. pb.paramEntityNames already knew it.", "file": "`mdl/executor/cmd_pages_builder_v3.go` (the \"association\" branch of buildDataSourceV3); tests `mdl/executor/cmd_pages_builder_assoc_pagelevel_test.go`; example `mdl-examples/bug-tests/assoc-datasource-page-level.mdl`", "insight": "The report's own diagnosis was the useful part and was right: it noticed the same path worked INSIDE a data view and failed at page level, which localises the bug to the context rather than to the association logic. Worth generalising \u2014 a resolver that takes an ambient context is correct exactly where the ambient context exists, and its fallback is what runs everywhere else; that fallback had a plausible comment (\"matches the common FROM=context pattern\") and was silently wrong for the whole page-level case. Two measurement notes: check-mdl cannot catch this at all, because it only runs `mxcli check` and the defect is in what the WRITER stores \u2014 the regression net had to be exec + mx check over the six examples that use an association datasource (five clean, the sixth failing identically before and after on an unrelated CE0106). And the reverse-direction control initially failed for the wrong reason: traversing a Reference from its FROM end yields ONE object, so a grid over it is [CE8812] \"A grid association path must result in a list\" \u2014 a cardinality complaint, not a resolution one, and the control had to become a data view to test what it claimed to test."} +{"area": "mdl/executor", "date": "2026-09-06", "symptom": "Every mxcli-authored Image widget failed the build. On a project at 0 errors, one `image imgProbe (Image: '...', Responsive: false)` on a new page gives [CE0463] \"The definition of this widget has changed\u2026\" at Image 'imgProbe'. A field-level diff of Atlas' brand image against a describe -> rename -> exec copy of it differs in ONE line of 1480: `maxHeight` = '0' where Atlas stores '250' (mxcli-ledger FINDINGS \u00a7142). A previous fix (ee295467) that named maxHeight explicitly was in the shipped binary and changed nothing.", "cause": "hiddenUnnamedProperties default-values a hidden property MDL cannot name, and maxHeight IS hidden (\"hidden when maxHeightUnit = none\"). Its condition property `maxHeightUnit` is unmapped too, so widgetValueMap does not know it, and the fallback read the DECLARED default \u2014 which Image 1.6.0 states as \"pixels\". Condition false -> maxHeight read as visible -> no reset -> the template's captured 0 stood. But maxHeightUnit being unmapped is exactly why the document gets the template's \"none\", not \"pixels\". Fixed by inserting the template's captured configuration (builder.PrimitiveValues(), read before any mapping is applied) between the script's values and the declared defaults in that fallback chain.", "file": "`mdl/executor/widget_engine.go` (hiddenUnnamedProperties' condition fallback + the Build call site); `mdl/backend/widgetobj/builder.go` (new PrimitiveValues, wrapping the existing primitiveValuesOf); `mdl/backend/mutation.go`, `mdl/backend/mcp/widget.go`; tests `mdl/executor/widget_hidden_reset_unmapped_test.go`; example `mdl-examples/bug-tests/image-widget-hidden-maxheight.mdl`", "insight": "The first fix passed a test built on an input that does not exist: its defaults helper declared maxHeightUnit's default as \"none\", and the real package declares \"pixels\" \u2014 the one value at which the rule fires. A fixture that encodes the value under test is not a fixture. The general shape: a visibility rule must be evaluated against the configuration that will be WRITTEN, never against the one the package declares, and the two diverge precisely on unmapped properties, which is the only place the rule matters. Two measurement traps cost time here. (1) `mxcli docker check` runs `mx update-widgets` first, which reconciles the widget and reports 0 errors while the stored value is still 0 \u2014 the same project reads 0 errors through it and 1 error through scripts/mx-check.sh. Use raw mx check for CE0463. (2) The ground truth was cheap and decisive once asked for: dumping all 69 Image widgets of a real project showed all 65 carrying a maxHeight store 250 at every combination of heightUnit/maxHeightUnit, and mxcli's was the sole outlier. Proven both ways by patching the stored 0 to 250 (1 error -> 0) and back (0 -> 1)."} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "describe navigation -> exec destroyed a menu item's glyph icon. Before: an item with Forms$GlyphIcon and a '-- not reproducible' comment. After exec of DESCRIBE's own output: no icon, no comment, exit 0, 'Navigation profile updated.'", "cause": "Mendix stores THREE icon elements (Forms$IconCollectionIcon{Image}, Forms$GlyphIcon{Code}, Forms$ImageIcon{Image}) and every layer handled only the first. The reader captured $Type and Image but never the glyph's Code, the AST/spec carried a bare name with no kind, and both writers emitted IconCollectionIcon unconditionally. Since CREATE OR REPLACE NAVIGATION is a full replacement, an icon the writer would not emit was an icon the statement DELETED.", "file": "mdl/executor/cmd_navigation.go", "insight": "The giveaway was already in the output and read as harmless: DESCRIBE printed '-- icon … is not reproducible by CREATE NAVIGATION; set it in Studio Pro'. That comment was written to make the loss VISIBLE, and it did — but only in the describe output, not in the exec that then acted on it. A note saying 'I cannot reproduce this' beside a FULL-REPLACEMENT statement is a note saying 'running this deletes it', and nobody made that connection. When a describer declines to emit something, check what the corresponding writer does with the omission. Fix shape: give each variant its own keyword (`icon glyph N` / `icon image QN`, bare = collection) so replay rebuilds the same ELEMENT — widening the bare form to cover all three would have converted an image icon into a collection icon, a silent variant swap. Also: preserve-when-silent was the right fix only while the construct was INEXPRESSIBLE; once authoring exists, omission becomes a real choice and preserving would make it impossible to remove a glyph icon.", "issue": "ako/mxcli"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "A brand-new lint rule (MDL074, 'menu item specifies no icon') would have fired on the majority of real menus, reporting items that plainly HAVE an icon.", "cause": "The rule tested `item.Icon == \"\"`. A Forms$GlyphIcon carries a numeric Code and NO qualified name, so the obvious emptiness test calls a perfectly good icon absent. The fixture's own Home item is glyph 57377.", "file": "mdl/executor/validate_navigation_icons.go", "insight": "Caught by writing the rule and the authoring support in the same session — the round-trip test emitted `icon glyph 57377` and check then warned about it, which is what exposed it. A rule that asks 'is this field empty' about a polymorphic element is asking the wrong question: ask the KIND. Generalisable: when a model element has variants with different payload shapes, any emptiness test on one variant's payload is a latent false positive on the others.", "issue": "ako/mxcli"} diff --git a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl index e0ed86139d..8bed9f5ccc 100644 --- a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl @@ -51,3 +51,5 @@ {"area": "mdl/grammar", "date": "2026-08-27", "raw": "| After ako/mxcli#260, ten of the 327 demo-app mappings still describe into MDL that does not parse — an export root printed as `. {`, and a custom-handler parameter printed as `Suggestion: (Value)` | Two leftovers of the same families. (1) `group as` covers a nested entity-less node (#262) but a ROOT has no member name, so it had no spelling at all. (2) `customHandlerParamText` rendered the stored value path raw, so an array-of-primitives parameter leaked Mendix's `(Wrapper)`/`(Value)` markers | `mdl/grammar/domains/MDLDomainModel.g4` (`exportMappingRootElement`), `mdl/executor/cmd_export_mappings.go` (the entity-less-root branch, and the association-less handling), `mdl/executor/mapping_customhandler.go` (`customHandlerParamText`, `buildCustomHandler`) | **Printing the member is only half of it**: emitting `Value` while the builder concatenates it back as `…|(Wrapper)|Value` gives a path that resolves to nothing, so resolve the parameter path THROUGH the schema index (`resolvePathKind(..., true)`) the way a member reference is resolved. Two things only a real build caught, both invisible to `mxcli check`: an entity-less root fell through to the VALUE branch and produced a project mxbuild cannot LOAD (*\"Type ExportValueMappingElement does not contain a constructor with a parameter of type ExportMapping\"*), because the builder decides object-vs-value on `def.Entity != \"\"`; and an element with NO association cannot be `Find` — **CE0224 \"No association selected for obtaining objects.\"** — it is `Parameter`, which is what CapitalConnector.EM_AttachedDataRequest stores on both its elements. Took the corpus from 317/327 parsing to **327/327**. Repro `mdl-examples/bug-tests/mapping-260b-last-parse-failures.mdl` |", "refs": ["#262", "ako/mxcli#260"], "ce": ["CE0224"]} {"area": "mdl/grammar", "date": "2026-08-28", "raw": "| A JSON structure's ARRAY ITEM element gets a derived name (`LinesItem`, `JsonObject`) that no MDL can change, and every mapping over the structure carries it — `describe` of a Studio Pro mapping then diffs on `ExposedName`. The obvious workaround, `custom name map ('lines\\|(Object)' as 'OrderLine')`, parses, executes and does **nothing** | An array's item is the anonymous `[…]` entry, so it has no JSON key; `customNameMap` is keyed on JSON keys, so the item was unreachable by construction. And an entry matching no key was applied to nothing and reported nothing, so the failed workaround was indistinguishable from success | grammar `mdl/grammar/domains/MDLDomainModel.g4` (`customNameMapping` gains `ITEM OF`), `mdl/ast/ast_jsonstructure.go` (`CustomItemNameMap`), `mdl/visitor/visitor_jsonstructure.go`, `mdl/types/json_utils.go` (`snippetBuilder.itemName`, `SnippetKeys`), `mdl/executor/cmd_jsonstructures.go` (`collectCustomItemNames`, DESCRIBE), `mdl/executor/validate_json_structure_names.go` (`MDL-JSON01`/`MDL-JSON02`) | **Do NOT infer Studio Pro's generation rule from stored documents — they are hand-edited.** Measured across 621 array elements in nine apps: 61% are the generator's `JsonObject`/`Wrapper`[+counter], 35% are a word someone chose, 4% are `Item`. The fingerprint that settles it is `JSON_AutoConfigResponse`, whose eight arrays read *in document order* `Scope, Wrapper_2..Wrapper_6, Claim, CodeChallengeMethods` — a counter with **gaps where a human renamed**; and `JSON_SensorData`'s `Array → SensorData`, a name with no relation to the array's. A version story fitted to a 9-structure sample (\"10.24 singularises, 11.4 does not\") evaporated at corpus scale. So the fix is **expressiveness, not default-matching**: no default can match a corpus that is a third hand-written, and changing the default would rewrite every stored structure's item names plus every mapping bound to one (ExposedName is a resolution key, #882). Design notes: `item of 'key' as 'Name'` rather than folding it into the existing entry, so naming an item does not require restating the array's name and adding one is a one-line diff; the same clause names a primitive array's **Wrapper**, because that wrapper IS the item; `item of 'Root'` for a root array, which has no key. DESCRIBE needs its own collector — an item's path segment is the marker `(Object)`/`(Wrapper)`, so the existing one skips it, and without it a named item was written on CREATE and silently renamed back by describe → exec. ako/mxcli#272 |", "refs": ["#882", "ako/mxcli#272"]} {"area": "mdl/grammar", "date": "2026-08-31", "symptom": "`DESCRIBE MICROFLOW` emits `reduce($list, expr)` (or `all(...)` / `any(...)`) and mxcli's own checker then rejects its own output: \"set 'X' calls 'reduce()', which is not a Mendix expression function [MDL044]\". Note the word **set** — the parser did not reject the call, it read the line as a Change Variable whose value happened to be a function call, and MDL044 was right about the rest", "cause": "DESCRIBE rendered an aggregate as `strings.ToLower(storedEnumValue)`, assuming every value of Mendix's `AggregateFunction` was also an MDL keyword. Mendix has eight, the grammar had five. Underneath sat a quieter defect: Mendix stores a Reduce's seed and result type in `ReduceInitialValueExpression` / `ReduceReturnDataType` and the semantic model had no field for either, so a grammar-only fix would have round-tripped the syntax while deleting the fold", "file": "`mdl/grammar/MDLLexer.g4` (REDUCE/ANY/INITIAL + the `keyword` rule so they stay usable as identifiers), `mdl/grammar/domains/MDLMicroflow.g4` (`listAggregateOperation` + `reduceFoldOptions`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `mdl/executor/cmd_microflows_builder_actions.go`, `mdl/executor/cmd_microflows_format_action.go` (`mdlAggregateKeyword`), plus all four read/write paths: `sdk/mpr/parser_microflow.go`, `sdk/mpr/writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go`, `mdl/backend/modelsdk/microflow_write.go`", "insight": "**A renderer that stringifies an enum outgrows its grammar silently** — the sibling `formatListOperation` switches on concrete types and cannot, which is the shape to prefer. The guard is a describe→parse loop over `microflows.AllAggregateFunctions` (`TestDescribedAggregateParsesBack`), so a ninth Mendix function fails a test rather than a user's script. **Get a reference document before believing the vendor docs**: Mendix's reference guide says a return type is \"not applicable\" to All/Any, but Studio Pro writes `ReduceReturnDataType` as Boolean on both, and `Attribute` as `\"\"` when unused — all three activities now re-serialize byte-identically to Studio Pro's. `mx check` is no help here (0 errors before and after); the controls are the origin/main parse (`reduce`/`all`/`any` → Change Variable, with `sum` → aggregate as the positive control) and reverting the write path (`TestReduceFoldReachesStorage` then reports the two keys missing). #1004", "refs": ["#1004"], "rules": ["MDL044"]} +{"area": "mdl/grammar", "date": "2026-09-04", "symptom": "`container c ()` / `dynamictext t ()` / `pluggablewidget 'id' pw ()` are parse errors, reported at the `)` as though the widget were wrong, while bare `container c` and `container c (x: 'y')` both parse.", "cause": "widgetPropertiesV3 was `LPAREN widgetPropertyV3 (COMMA widgetPropertyV3)* RPAREN` — at least one property required.", "file": "mdl/grammar/domains/MDLPage.g4", "insight": "An empty property list is what an LLM writes for a widget that needs no properties, and the error points at the paren rather than the cause. One-character fix (wrap the list in `( … )?`). Found while measuring something else — the first run of a keyword-parse survey used `()` throughout and mis-scored every keyword as rejected, including ones that worked. If a whole measurement comes back uniformly negative, suspect the harness before the subject.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/grammar", "date": "2026-09-05", "symptom": "After adding a generic (IDENTIFIER | keyword) alternative to widgetTypeV3, `slot body` parsed as a widget of type `slot` and `placeholder Main { ... }` as a widget named Main. Both still parsed, `mxcli check` still exited 0, and a diff of check output across all 515 mdl-examples scripts showed ZERO difference.", "cause": "pageBodyV3 listed widgetV3 FIRST, before useFragmentRef / placeholderBlockV3 / slotMarkerV3. SLOT, PLACEHOLDER and USE are all inside the `keyword` rule (655 tokens), so the generic widget alternative matched them before the specific alternative could.", "file": "mdl/grammar/domains/MDLPage.g4", "insight": "Put the specific alternatives BEFORE widgetV3 in pageBodyV3, the same ordering fix widgetV3 already applies internally for `template for`. The transferable lesson is about the MEASUREMENT, not the grammar: a corpus diff of `mxcli check` output compares DIAGNOSTICS, so it is blind to a construct that parses into the wrong AST shape. 515 scripts said nothing; two visitor unit tests caught it immediately. When a grammar change could reinterpret existing syntax rather than reject it, assert on the AST, not on the diagnostics. CONTROL: reorder pageBodyV3 back and TestSpecificPageBodyFormsWinOverTheGenericWidget / TestSlotMarkerWinsOverTheGenericWidget fail.", "issue": "mendixlabs/mxcli#1036"} diff --git a/.claude/skills/fix-issue/findings/mdl-other.jsonl b/.claude/skills/fix-issue/findings/mdl-other.jsonl index afdf7116b0..e4aa5b8b03 100644 --- a/.claude/skills/fix-issue/findings/mdl-other.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-other.jsonl @@ -52,3 +52,9 @@ {"area": "mdl/translations", "date": "2026-08-30", "raw": "| `create or modify translations in for ` reports success (\"Set 212 nl_NL translation(s) across 20 document(s)\") and the app's pages switch language while the **menu does not** | `mdl/translations/outofscope.go` (new), `mdl/executor/cmd_translations.go`, `cmd/mxcli/syntax/features_misc.go` | The **navigation is a project-level document**, not a module one, so `in ` never reaches it. Measured on the reporting project: 151 strings scoped against 546 unscoped, and re-running the same file unscoped landed 65 more across 22 further documents. Nothing warned — the document count was the only tell, and only if you knew what number to expect. A scoped run now names **the file's own entries** it did not reach (`translations.OutOfScope`), not \"the project has other strings\", which is true of every scoped run and would warn forever — the per-module workflow is exactly what the scoping exists to support. Second, load-bearing half: those entries were previously swept into the **drift** warning, whose premise (\"no text has this as its source\") is *false* about them — they matched, out of scope. They are subtracted from it, so \"the text may have been deleted\" is only said where it is true. Controls: an unscoped run of the same file reports nothing new and lands the strings; a key matching nothing anywhere is still reported as drift. Reported as ledger #137 |", "refs": ["#137"]} {"area": "mdl/catalog", "date": "2026-09-03", "symptom": "`SHOW LANGUAGES` omits a language the project really has (ar_DZ absent from a list of 8 where the project has 9), and `search ''` returns \"No matches found\" for a string `DESCRIBE TRANSLATIONS` lists. Nothing errors and the catalog builds clean.", "cause": "CATALOG.strings was filled by hand-written per-type extractors reaching five sites (page title, enum caption, three microflow message templates), so a text anywhere else — every widget caption, tooltip, validation message, client template — was never indexed.", "file": "mdl/catalog/builder_strings.go", "insight": "A language present only on an unindexed site is INVISIBLE, not undercounted, so it vanishes from SHOW LANGUAGES entirely and from lint rule QUAL005, which discovers its language set from the same table. The fix is not a sixth case — that is how five was ever the number. Index from the type-agnostic walk DESCRIBE TRANSLATIONS already uses (translations.SitesInUnit over ListRawUnitsByType(\"\")), leaving only non-Texts$Text strings in the typed path (URLs, log nodes, REST paths, documentation, and Microflows$StringTemplate, which holds a plain Text and cannot carry a translation). Derive ObjectType from the unit $Type mechanically rather than via a table. Measured before: 69 of 3265 texts, 8 of 9 languages, 66 en_US of 1045. After: 1496 rows, 9 languages, counts identical to an independent BSON walk. Atlas design templates are ~70% of the corpus and are indexed rather than excluded, because CREATE TRANSLATIONS writes them and a SHOW LANGUAGES that excluded them would reopen the same split. CONTROL: stub the walk and the run reports `strings: 3` with SHOW LANGUAGES reporting nothing at all.", "refs": ["#250"]} {"area": "mdl/linter", "date": "2026-09-03", "symptom": "Lint rule QUAL005 reports no missing translation for an enumeration where only one value is translated (11 real gaps unreported), and likewise for a page's sibling action buttons.", "cause": "The rule grouped by (QualifiedName, StringContext) while ElementId sat unused in the strings table, so every sibling element of one type collapsed into one group and a single translated value made the set look complete.", "file": "mdl/linter/rules/missing_translations.go", "insight": "Add ElementId to the SELECT, the ORDER BY and the elementKey struct. No test caught it because the harness synthesized ElementId from QualifiedName+StringContext, giving every sibling the same value and reproducing the defect inside the fixture — a fixture that encodes the bug cannot detect it. CONTROL: with every sibling translated the run must stay at 0 violations, or the new violation is an artifact of splitting the group rather than the missing translation.", "refs": ["#250"]} +{"area": "mdl/catalog", "date": "2026-09-05", "symptom": "A widget inside a DataGrid2 column, gallery item or chart series was absent from CATALOG.WIDGETS, so a page holding 19 chart sparklines in datagrid columns never appeared under \"which pages use VegaChart?\" while the grid around them did.", "cause": "extractWidgetsRecursive walked a pluggable widget's Object.Properties[].Value.Widgets (a child slot) but not Value.Objects[] (an object list), whose items are themselves property bags holding widgets.", "file": "mdl/catalog/builder_pages.go", "insight": "Wider than the widget edge: CATALOG.REFS is a projection of this table, so an entity or microflow used ONLY inside a column template reported zero references — anything using reference counts to decide 'unused, safe to delete' would delete a document in active use (#940's failure mode, fixed for List View templates and left open for object lists). Recurse (an item is a property bag) rather than special-casing a depth. TWO measurement traps: the unfiltered ref count GREW when the widget edge landed, and growth looks like progress — only a query filtered to the widget actually asked about shows the gap; and .mxcli/catalog.db must be DELETED between runs, since a stale cache made the fixed binary look unfixed and produced a wrong conclusion before the rebuild was forced.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/exprcheck", "date": "2026-09-05", "symptom": "A non-String log-message template parameter passed `check --references`, passed `exec`, and failed the build: `LOG WARNING 'qty {1}' WITH ({1} = $Order/Qty)` gave [CE0117] \"Error(s) in expression.\" at the Log message activity (mendixlabs/mxcli#1043).", "cause": "TWO gaps in series. The adapter walked LogStmt.Message and never the template parameters, so they were not checked at all; and exprcheck's slot-expectation table had existed with NOTHING READING IT \u2014 slotKind() was defined and never called \u2014 so even the pre-existing `LogStmt.Message: {Kind: KindString}` constrained nothing and `LOG WARNING 42` passed too. Wired checkSlotKind at the top of Parse, added the LogStmt.TemplateParam slot, and taught inferKind that a variable the entity scope knows holds an Object.", "file": "`mdl/exprcheck/parser.go` (checkSlotKind, inferKind), `mdl/exprcheck/slot_resolver.go`, `mdl/exprcheck/adapters/check.go`; tests `mdl/exprcheck/slot_kind_test.go`", "insight": "A declared-but-unread table is worse than a missing one: it reads as coverage in review and in the source, and every entry in it is a rule someone believed was enforced. Finding it took noticing that the FIX did not work \u2014 adding the new slot changed nothing \u2014 rather than that the bug existed. The report's diagnosis was also wrong in a way worth recording: it generalised to 'the writer's template/parameter emission is wrong, not the caller's type', but toString(...) around the identical value builds cleanly, so the writer is fine and Mendix simply does not coerce. Measured which kinds actually fail rather than assuming 'objects': four microflows executed and built gave three CE0117s \u2014 String clean, Integer/Decimal/Boolean/DateTime/Object all failing. Enforcement is deliberately limited to CONCRETE expectations; an entry carrying ResolveBy names a kind resolved per call site, and the adapter encodes that by appending the target to the slot path, so those paths do not match the table at all and enforcing them would compare against a placeholder kind."} +{"area": "mdl/exprcheck", "date": "2026-09-05", "symptom": "A bare word as a member's value passed check and exec and failed the build: `CHANGE $Order (Status = Closed)` gave [CE0117] \"Error(s) in expression.\" at the Change object activity (mendixlabs/mxcli#1044).", "cause": "Mendix expressions have no bare identifiers, but the parser reads one as a variable reference (parseIdentLed's fallthrough returns VariableExpr). It then resolves to nothing, infers KindUnknown, and Unknown is tolerated by every rule by design. Added a Bare flag to VariableExpr, set where the node is built from a plain TokIdent rather than a TokDollarIdent, and a rule (E013) that reports one standing alone as a create/change member's value.", "file": "`mdl/exprcheck/parser.go` (checkBareIdentifierValue, parseIdentLed), `mdl/exprcheck/ast.go`, `mdl/exprcheck/hints/registry.go`", "insight": "The scoping is the whole design, not caution. A bare name NESTED in a list-operation predicate is legal MDL \u2014 `FILTER($L, Status = 'Open')` resolves `Status` against the item under test \u2014 so the obvious rule ('no bare identifiers in expressions') rejects working scripts, and the report's own second example is one of them. A member's value is the one position where the bare word is the ENTIRE expression and can only be a mistake, which is why the rule keys on the slot path and the top-level node rather than on the token. Consequence worth stating: the report's `FILTER($AllOrders, Status = Open)` case is NOT covered, because the bad half is nested; catching it needs knowing which side of a comparison is the attribute."} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "An unqualified CREATE passed `check --references` and was refused by exec: `create association Order_Probe from Bench.Order to Bench.Customer` -> \"module name is required: objects must be created within a module\" (mendixlabs/mxcli#1050). Separately, `RETURNS void AS $result` passed check and wrote `return $result` into a flow with no such variable -> [CE0109] \"Undefined variable 'result'.\" at End event (mendixlabs/mxcli#1041).", "cause": "Neither had a check-time rule. Added MDL074 (ValidateCreateIsQualified, driven off the existing stmtCreateInfo walker so it covers every document type rather than just associations, with CreateModuleStmt excluded) and MDL075 (ValidateVoidReturnAlias), both in the NO-PROJECT pass since both are answerable from the statement alone.", "file": "`mdl/executor/validate_create_shape.go` (new), `mdl/executor/validate_program.go` (wiring); tests `mdl/executor/validate_create_shape_test.go`", "insight": "The reportable cost of an unqualified CREATE is not the error message, it is that exec is NOT TRANSACTIONAL: the statements before it are already applied, so 're-run it' then fails with 'already exists' on those. That is what makes a check-time rule worth more than a better exec error. For the void alias, refusing beats repairing: emitting a bare `return` would also build, but an author who wrote an alias meant to return something, and silently dropping it leaves the flow returning nothing while the source still says otherwise. Reusing stmtCreateInfo was the leverage \u2014 validate_association_module.go had already documented this exact gap in a comment ('check does pass it, which is a check/exec divergence worth closing on its own') and closing it association-only would have left every other document type open."} +{"area": "mdl/exprcheck", "date": "2026-09-05", "symptom": "`IF empty($Orders)` was rejected with a hint carrying NO code (printed as `[]`), no document and no microflow, and a fix line that named only glued keywords ('emptyor') \u2014 so the reader had to find the expression in a 40-line script by eye and was pointed at a cause that was not there (mendixlabs/mxcli#1042).", "cause": "The trailing-token hint in Parse was built by hand with a bare hints.Location holding only line/column, rather than through hintsLocation(ctx, pos) which carries File and Microflow. Gave it code E014, a registry entry, hintsLocation, and a fix line naming the real cause first: `empty` is a Mendix KEYWORD, not a function, so the parser consumed it and stopped at the '('.", "file": "`mdl/exprcheck/parser.go` (the trailing-token hint), `mdl/exprcheck/hints/registry.go`", "insight": "The line and column are offsets into the EXPRESSION FRAGMENT, not into the file, so they cannot be made into a file position without the adapter setting Context.File/Line \u2014 the microflow name is the achievable locator and the example says so rather than implying more. Separately, a tier-ordering fact worth knowing when reproducing any E0xx: the reference check runs BEFORE expression checking and exits on its first error, so one unrelated mistake anywhere in a file hides every expression hint in it. That cost real time here \u2014 a control microflow in the new example had `set $out = \u2026` without declaring $out, and the resulting reference error made the E014 perturbation look like it was not firing at all. I guessed at two wrong explanations (needs no project; duplicate-CREATE conflict) before bisecting."} +{"area": "mdl/linter", "date": "2026-08-31", "symptom": "`mxcli lint` reports a nanoflow or a rule as a microflow — \"Microflow 'Rule1' has no activities [MPR002]\" about a rule, \"Microflow 'Nanoflow' …\" about a nanoflow. The same wrong noun reaches the JSON and SARIF `documentType` field, where it is not merely cosmetic", "cause": "Microflows, nanoflows and rules share one catalog table (`microflows`, discriminated by `MicroflowType`; the `nanoflows` and `rules` views are filters over it), so `LintContext.Microflows()` yields all three. Eleven rule call sites hardcoded `DocumentType: \"microflow\"`, four of them also putting the word in the message", "file": "`mdl/linter/context.go` (`Microflow.DocumentNoun`/`DocumentNounTitle`), then the call sites in `mdl/linter/rules/`: `empty.go` (MPR002), `conv_loop_commit.go`, `conv_error_handling.go`, `conv_split_caption.go`, `flow_irreducible_graph.go`, `mpr008_overlapping_activities.go`, `mpr011_loop_child_containment.go`, `naming.go`, `validation_feedback.go`", "insight": "**A shared table means a shared iterator, and an iterator yielding three doctypes needs the noun derived, never literal.** Grep `ctx.Microflows()` before assuming a rule is microflow-only — nine rules use it. Fix the labelling only; which documents a rule applies to is a separate question from what the report calls them. Two traps: widening a format string is **not** a compile error (`fmt.Sprintf` is variadic), so `go vet` is the gate that catches the missing argument, not `go build` — it caught two here. And the pre-existing fixtures in `empty_test.go` spell the type title-case (`\"Microflow\"`) where the catalog writes uppercase (`\"MICROFLOW\"`, `mdl/catalog/builder_microflows.go`); harmless while nothing read the column, but a fixture with the wrong case now exercises the fallback instead of the mapping. Control: revert the two lines in `empty.go` and `TestEmptyMicroflowRule_NamesTheDocumentType` reports the symptom verbatim. `mx check` and the build are irrelevant — this is mxcli's own output, not the model", "rules": ["MPR002"]} diff --git a/.claude/skills/mendix/check-syntax/SKILL.md b/.claude/skills/mendix/check-syntax/SKILL.md index 9989cb3906..393e0cb139 100644 --- a/.claude/skills/mendix/check-syntax/SKILL.md +++ b/.claude/skills/mendix/check-syntax/SKILL.md @@ -54,6 +54,95 @@ icon or entity sailed through a command that had been handed the project. A run without a project now says what it did not check, so a pass is never read as more than it is. +### It resolves MEMBER names too, where it can establish the entity + +Resolution does not stop at the entity. An attribute named in a **create** or +**change** activity is looked up on that entity *and its generalizations*, so a +typo is reported by `check` rather than by mxbuild as `CE1613 "The selected +attribute '…' no longer exists"` a whole build later: + +``` +Sales.ACT_Close: Sales.Order has no member "IsArchived" (in change $Order) + — it has OrderNo, Status — mxbuild reports this as CE1613 … +``` + +This needs the target's entity to be **known**, and that is the boundary worth +understanding rather than assuming: + +| the object comes from | checked? | +|---|---| +| a `create Module.Entity (…)` | yes — the entity is in the statement | +| a microflow/nanoflow **parameter** | yes | +| `retrieve $L from Module.Entity` | yes | +| `retrieve $L from $Obj/Module.Assoc` | yes, when `$Obj` is itself typed | +| a `loop` over any of those | yes — the iterator inherits the element type | +| anything else (`send rest request`, `response: file as $Doc`, …) | **no** | + +Widget positions are resolved too: + +- an **XPath constraint** on a `database from Module.Entity` source — every step + is followed, so a bare name must be an attribute of the entity it lands on and + a `Module.Name` step must be an association or an entity; +- a **template parameter** (`ContentParams` / `CaptionParams`) rooted in a + variable. That one needs **no project** and fires under a bare + `mxcli check`, because the answer is in the statement. + +The template-parameter rule is narrower than "no `$` roots", and the difference +is measured rather than reasoned — the writer strips one prefix on one branch: + +| `{1} = …` | | +|---|---| +| `OrderNo` | fine | +| `Order_Customer/Name` | fine — association hop, then attribute | +| `$currentObject/Order_Customer/Name` | fine — the prefix is stripped | +| `$currentObject/OrderNo` | **CE1613** | +| `$Order/Name` | **CE1613** | + +Note the two-segment form: `Assoc/Attr`, not the XPath `Assoc/Entity/Attr`, +which mxbuild also rejects. + +### Expression KINDS are checked in the positions that declare one + +Two more things reach mxbuild as `CE0117 "Error(s) in expression"` and are now +reported by `check`: + +- **A bare word as a member's value.** Mendix expressions have no bare + identifiers, so `CHANGE $Order (Status = Closed)` is E013. Write `'Closed'` + (a literal), `$Closed` (a variable), or `Module.Enum.Value` (an enumeration). + Scoped to the *whole* value of a create/change member: a bare name **nested** + in a list-operation predicate is legal — `FILTER($L, Status = 'Open')` + resolves `Status` against the item under test — and is not reported. +- **A log message's template parameter must be a String.** `LOG … WITH ({1} = + $Order/Qty)` is E009. Measured on 11.13.0: Integer, Decimal, Boolean, + DateTime and an object each fail; a String attribute is clean; and + `toString(…)` around any of them is clean. So wrap the non-String ones — + the writer is fine, Mendix simply does not coerce here. + +### Three more things check now refuses + +- **An unqualified CREATE** (`create association Order_Probe …`) — MDL074, no + project needed. `exec` always refused it; check now does too, which matters + because exec is **not transactional**: the statements before the failure are + already applied, and re-running hits "already exists" on them. +- **`RETURNS void AS $x`** — MDL075, no project needed. An alias names the + variable a flow returns, so it cannot be paired with void; mxcli used to + believe the alias and write `return $x` into a flow with no such variable + (CE0109). Write `RETURNS void`, or give the alias the type it holds. +- **`empty($List)`** — E014. `empty` is a Mendix **keyword**, not a function, so + the parser stops at the `(`. Write `$List = empty` or `length($List) = 0`. + +**One thing to know about hint ordering**: the reference check runs before +expression checking and **exits on its first error**, so an unrelated mistake +anywhere in a file hides every expression hint in it. If you expect an E0xx and +see none, fix the reference errors first and re-run. + +A variable this cannot type is left **unchecked**, never guessed at — a false +"no such member" would block a script that builds cleanly. Two more things are +deliberately not reported: a **qualified** member (`Module.Assoc`), which exec +already refuses when it cannot be an attribute, and any member on an entity the +**script itself** creates or whose attribute the script adds earlier — the +add-the-column-then-populate-it shape stays valid. + ## Pre-Flight Validation Checklist Before writing any MDL, verify these requirements: diff --git a/.claude/skills/mendix/create-page/reference/widgets.md b/.claude/skills/mendix/create-page/reference/widgets.md index 33f48bd736..4a462da8c3 100644 --- a/.claude/skills/mendix/create-page/reference/widgets.md +++ b/.claude/skills/mendix/create-page/reference/widgets.md @@ -865,9 +865,9 @@ create page Sales.Dashboard (Title: 'Revenue', Layout: Atlas_Core.Atlas_Default) series sRevenue ( dataSet: static, DataSource: database from Sales.ByRegion, -- or: staticDataSource: database from Sales.ByRegion - StaticXAttribute: Region, - StaticYAttribute: Total, - StaticName: 'Revenue' + staticXAttribute: Region, + staticYAttribute: Total, + staticName: 'Revenue' ) } } diff --git a/.claude/skills/mendix/custom-widgets/SKILL.md b/.claude/skills/mendix/custom-widgets/SKILL.md index 0c4bc76f4d..9b319361ad 100644 --- a/.claude/skills/mendix/custom-widgets/SKILL.md +++ b/.claude/skills/mendix/custom-widgets/SKILL.md @@ -1,10 +1,65 @@ --- name: custom-widgets -description: "MDL syntax for pluggable widgets in CREATE PAGE / ALTER PAGE — GALLERY, COMBOBOX, DataGrid2 and third-party widgets: datasource and column forms, child slots (TEMPLATE/FILTER), adding a widget via .def.json, and the engine internals. Use when placing a pluggable widget on a page, or when `mxcli widget describe` output needs interpreting. For the widgets THIS project actually has, read the generated `widgets` skill." +description: "MDL syntax for pluggable widgets in CREATE PAGE / ALTER PAGE — any installed widget is named by its own name (`htmlelement frame (…) { … }`), with object lists and child slots read from its definition. Covers GALLERY, COMBOBOX, DataGrid2, charts and third-party widgets: datasource and column forms, child slots (TEMPLATE/FILTER), the `pluggablewidget ''` fallback, and adding a widget via .def.json. Use when placing a pluggable widget on a page, or when `mxcli widget describe` output needs interpreting. For the widgets THIS project has, read the generated `widgets` skill." --- # Custom & Pluggable Widgets in MDL +## Any installed widget is named by its own name + +If a widget is installed in `widgets/`, MDL names it directly — no keyword list, +no widget id: + +```sql +htmlelement frame (tagName: 'div', tagContentMode: 'container') { + attribute a1 (attributeName: 'data-testid', attributeValueType: 'expression') + tagcontentcontainer body { + dynamictext caption (Content: 'Inside the element') + } +} +``` + +Three things there are read from the widget's definition, not from anything +hardcoded: the **keyword** (`htmlelement`, the last segment of the widget id), +the **properties** (the widget's own spelling — `tagName`, not `TagName`), and +the **body containers** — `attribute` is an object list (one entry per +repetition), `tagcontentcontainer` a child slot (holds widgets). + +**Ask the widget rather than guessing.** `describe widget ` lists every +property with its type, default and enumeration members; every body container +and whether MDL can express it; and a complete example that parses AND checks as +written: + +```bash +mxcli widget describe htmlelement -p app.mpr +``` + +Do this first when placing an unfamiliar widget. It is faster than reading this +file and it cannot go stale, because it reads the `.mpk` the project actually +has. + +### The id form is the fallback + +```sql +pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' frame (tagName: 'div') +``` + +Use it only when two installed packages ship the same MDL name, or when you have +the id and not the name. Everything below that still shows the id form works +unchanged — the short form is simply the better default. + +### When the name is not found + +A name resolving to no installed definition is an **error** (MDL-WIDGET25, with +near-miss suggestions), and a container the parent does not declare is +MDL-WIDGET26. Both need `-p`: without a project, mxcli knows only its embedded +widgets, so it stays quiet rather than reporting every real widget as unknown. +If a widget you have installed is not found, extract its definition: + +```bash +mxcli widget init -p app.mpr +``` + ## Built-in Pluggable Widgets ### GALLERY @@ -61,9 +116,14 @@ combobox cmbCustomer ( ## Charts (Mendix Charts.mpk) -Charts are pluggable widgets authored by their **package id**. Install `Charts.mpk` -into the project's `widgets/` folder first (any Charts-based app has it); `exec` -auto-generates the `.def.json`. +Charts are pluggable widgets. Install `Charts.mpk` into the project's `widgets/` +folder first (any Charts-based app has it); `exec` auto-generates the +`.def.json`. + +Each is authorable by its **own name** — `barchart`, `linechart`, `piechart`, +`heatmap` — and the examples below use the package id form, which also still +works. The id column is kept because it is what `describe widget` prints and +what identifies the widget unambiguously. **Chart type → widget id → data container:** @@ -79,12 +139,12 @@ auto-generates the `.def.json`. ``` pluggablewidget 'com.mendix.widget.web.barchart.BarChart' chart1 { series s1 ( - DataSet: 'static', + dataSet: 'static', DataSource: database from MyModule.SalesByRegion, -- an OQL VIEW (aggregated) - StaticXAttribute: Region, -- resolves against the series' own datasource - StaticYAttribute: Total, - StaticName: 'Revenue', - Interpolation: 'linear' -- line/area only: linear | smooth + staticXAttribute: Region, -- resolves against the series' own datasource + staticYAttribute: Total, + staticName: 'Revenue', + interpolation: 'linear' -- line/area only: linear | spline ) } ``` @@ -101,15 +161,15 @@ from`, so a microflow-backed series described back as a missing entity.) pluggablewidget 'com.mendix.widget.web.piechart.PieChart' pie1 ( DataSource: database from MyModule.SalesByRegion, ValueAttribute: Total, - SeriesName: 'Sales by Region' -- REQUIRED (CE4899 without it) + seriesName: 'Sales by Region' -- REQUIRED (CE4899 without it) ) pluggablewidget 'com.mendix.widget.web.heatmap.HeatMap' heat1 ( DataSource: database from MyModule.SalesByRegion, ValueAttribute: Total -- REQUIRED (CE0642 without it) ) { - scalecolor scLow (ValuePercentage: 0, ColorValue: '#f7fbff') - scalecolor scHigh (ValuePercentage: 100, ColorValue: '#08306b') + scalecolor scLow (valuePercentage: 0, colorValue: '#f7fbff') + scalecolor scHigh (valuePercentage: 100, colorValue: '#08306b') } ``` diff --git a/.claude/skills/mendix/generate-domain-model/reference/syntax.md b/.claude/skills/mendix/generate-domain-model/reference/syntax.md index 1fcea2abe8..f4f427b174 100644 --- a/.claude/skills/mendix/generate-domain-model/reference/syntax.md +++ b/.claude/skills/mendix/generate-domain-model/reference/syntax.md @@ -90,6 +90,19 @@ alter enumeration Module.TransactionType modify value EXPENSE caption 'Expense / alter enumeration Module.TransactionType drop value REFUND; ``` +**Add `if not exists` in a script you will re-run.** The bare `add value` errors +on the second run, and `exec` stops at the failing statement — so one +already-present value leaves every later statement in the file unapplied. The +same pair as `add attribute` / `add index`: + +```sql +alter enumeration Module.TransactionType add value if not exists REFUND caption 'Refund'; +alter enumeration Module.TransactionType drop value if exists REFUND; +``` + +A defensive drop-then-add is not a substitute: the drop fails when the value is +absent and the add when it is present. + `modify value … caption` re-captions in place — the value keeps its identity, so it works even while the enumeration is in use. (Value names in `alter` must be plain identifiers; a value whose name is a reserved word can't be targeted by `alter`.) diff --git a/.claude/skills/mendix/json-structures-and-mappings/SKILL.md b/.claude/skills/mendix/json-structures-and-mappings/SKILL.md index 300767fe67..48558f26e3 100644 --- a/.claude/skills/mendix/json-structures-and-mappings/SKILL.md +++ b/.claude/skills/mendix/json-structures-and-mappings/SKILL.md @@ -172,8 +172,9 @@ create message definition collection Sales.MD_Order ( ``` A bare name is an attribute; `Assoc/Module.Entity` is an association. **Name the -target entity** — the stored cardinality follows the direction of traversal, so -the same association gives a single object one way and a list the other. +target entity** — the stored cardinality follows the direction of traversal and +the association's type, so a `Reference` gives a single object one way and a list +the other, while a `ReferenceSet` is a list both ways. The full vocabulary, the ALTER statements, inherited attributes and what mxcli deliberately does not guess: diff --git a/.claude/skills/mendix/json-structures-and-mappings/reference/message-definitions.md b/.claude/skills/mendix/json-structures-and-mappings/reference/message-definitions.md index 99ddcc4f12..7db5877958 100644 --- a/.claude/skills/mendix/json-structures-and-mappings/reference/message-definitions.md +++ b/.claude/skills/mendix/json-structures-and-mappings/reference/message-definitions.md @@ -25,12 +25,26 @@ its own members — the same discriminator import and export mappings use. A mapping then binds to `Module.Collection.Definition`, a three-part reference. **Name the association's target entity.** It is not decoration: the stored -cardinality tracks the **direction of traversal**, not the association's type. +cardinality — single object or list — is derived from the **direction of +traversal** and the **association's type** together. + +| | forward (from the FK owner) | reverse | +|------------------|-----------------------------|--------------------| +| **Reference** | single object | list | +| **ReferenceSet** | list | list | + Reaching `Customer` from `Order` follows the foreign key and gives a single object; reaching `Order` from `Customer` is the reverse and gives a list — the -same association, both ways. An association that connects the two entities in -neither direction is **refused**, because a wrong cardinality builds cleanly and -would silently expose a list as a single object. +same association, both ways. A **ReferenceSet is a list in both directions**, +because a set is many at both ends. + +An association that connects the two entities in neither direction is +**refused**, because mxcli would have to guess. Guessing is worse than +refusing in opposite ways on the two halves of the rule, which is worth knowing +when something looks wrong: a wrong *direction* builds cleanly and silently +exposes a list as a single object, while a wrong *type* is caught — mxbuild +reports CE6524 (`The occurrence of '...' has changed`) on the definition and +CE0295 (`Association '...' is not allowed`) on any mapping element bound to it. **Inherited attributes are named like the entity's own.** mxcli resolves each to the entity that declares it, which is what Mendix stores; qualifying one against @@ -53,12 +67,31 @@ alter message definition collection Sales.MD_Order rename definition Line to Ord alter message definition collection Sales.MD_Order drop definition if exists OrderLine; ``` -`in ` reaches a nested member, written in **exposed names**. `SET` changes -only the exposed name — it is not a model rename, which is why the verb is not -`RENAME`. +`in ` reaches a nested member, written in **exposed names**. `DROP MEMBER` +takes the member's **original** name, though — the attribute's, or for an +association the target entity's — so the two halves of `drop member Tag in +Orders` are named differently on purpose. + +`SET` changes only the exposed name — it is not a model rename, which is why the +verb is not `RENAME`. + +### Dropping something a definition still needs + +A definition is a selection over the domain model held **by qualified name**, and +nothing keeps the two in step. Both directions are refused rather than left to +mxbuild: + +- **A definition a mapping still references** — refused, naming the mappings. +- **An association a definition still exposes** — refused, printing the `alter … + drop member` statement for each definition that exposes it, ready to run. All + of them are listed: an association exposed in both directions dangles from the + other one if you clear only the first. -Dropping or renaming a definition a mapping still references is refused, naming -the mappings. +Without the second, `drop association` reported success and mxbuild reported +CE1613 at the definition — and `describe` went on emitting the dangling member, +so the break survived a describe → exec round trip. The guard covers message +definitions only: a microflow retrieve or an object mapping element over the same +association is still your own CE1613 to resolve. ### What mxcli does not guess diff --git a/.claude/skills/mendix/manage-navigation/SKILL.md b/.claude/skills/mendix/manage-navigation/SKILL.md index 133217365d..e87d482b1f 100644 --- a/.claude/skills/mendix/manage-navigation/SKILL.md +++ b/.claude/skills/mendix/manage-navigation/SKILL.md @@ -118,9 +118,30 @@ create or replace navigation Responsive ### Menu Icons -Both `menu item` and `menu 'caption' (...)` take an optional `icon`. It is a -**qualified name** into an **icon collection** — a model reference, written like -every other reference in MDL, not a string: +**Give every menu item an icon.** It is optional in the grammar and `mxcli check` +warns when it is missing (**MDL074**), because the navigation sidebar collapses +to an icon rail and that is the state most users leave it in: a collapsed item +shows its icon, and one without falls back to the first few characters of its +caption — rarely enough to tell `Orders` from `Order lines`. Nothing else catches +it. The model builds, `mx check` passes, and the menu is simply hard to use. + +Both `menu item` and `menu 'caption' (...)` take an `icon`, in one of three +forms — Mendix stores three different icon **elements**, not three spellings of +one value: + +```sql +menu item 'Home' page M.Home icon Atlas_Core.Atlas.home; -- icon collection +menu item 'Close' page M.Close icon glyph 57377; -- numeric glyph code +menu item 'Logo' page M.Logo icon image M.Images.logo; -- image collection +``` + +The **bare** form is the icon-collection icon and is what you normally want. Use +`glyph` only to reproduce a legacy icon a project already has — `describe +navigation` emits it for you — and `image` for a picture from an image +collection, which is a different document from an icon collection. + +The icon-collection form is a **qualified name** — a model reference, written +like every other reference in MDL, not a string: ```sql create or replace navigation Responsive diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index 63e0a9431c..798b79324c 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -248,6 +248,33 @@ Mendix **CE0066** "Entity access is out of date", which masks the CE2729 "No read access to attribute" errors underneath until Studio Pro's *Update security* is clicked. +### `UPDATE SECURITY` — reconciling rules that have gone stale + +mxcli reconciles an entity's access rules whenever it writes the entity, so a +rule normally cannot go stale through mxcli. `UPDATE SECURITY` is the repair for +one that did — a model edited elsewhere, or an older mxcli: + +```sql +update security; -- every module the project owns +update security RestLab; -- one module (IN is optional) +update security in RestLab; -- the same thing +``` + +Three things worth knowing, each of which was a defect until +[mendixlabs/mxcli#1047](https://github.com/mendixlabs/mxcli/issues/1047): + +- **`System` is skipped.** Its entities are the platform's, its access rules are + Mendix's rather than the project's, and its domain model is not stored in the + `.mpr` at all. Naming it explicitly is an error, not a silent no-op. +- **One unreadable module does not end the run.** It is reported and stepped + over, so the rest are still reconciled. Previously the first failure returned, + and System guaranteed one on every project — which made the command inert. +- **The scope is honoured.** `update security RestLab` used to parse cleanly and + run project-wide, because the name reached the parser's error recovery. + +A run that skipped something says so. `All entity access rules are up to date` +means every module was looked at. + A member name that matches nothing is now an error rather than a silent skip: ``` diff --git a/.claude/skills/mendix/record-narrated-demo/SKILL.md b/.claude/skills/mendix/record-narrated-demo/SKILL.md index 5b228967a4..e781bf83a2 100644 --- a/.claude/skills/mendix/record-narrated-demo/SKILL.md +++ b/.claude/skills/mendix/record-narrated-demo/SKILL.md @@ -48,6 +48,25 @@ other: `INVALID` means the instrument did not run, which is a finding of its own **Writing the journey first is what makes the demo cheap.** The persona, the path, and the definition of success already exist by the time you record. +## Seed the data before you record + +The failure this skill is most prone to, and it is not subtle. The first real +ContactBook capture was an empty grid reading `0 to 0 of 0` with a column header +rendering as `colActions` — which reads as **a broken app, not a new one**. The +journey passed; the app was live; the recording was worthless. + +Before any take: + +- **Populate every list the camera will see**, with plausible values — real names + and amounts, never `test1` / `asdf` / `aaa`. +- **Caption every column and button the camera will see.** A header showing its + attribute name is the single clearest tell that nobody looked at the screen. +- **Open on something already interesting.** The first thing on screen should be + a populated state, not an empty one waiting to be filled. + +`mxcli` seeds this itself — see [demo-data](../demo-data/SKILL.md). Seeding is +part of recording, not a nicety before it. + ## Recording mechanics Four things that decide whether the video is watchable. Each has a reason; none @@ -59,6 +78,17 @@ A cursor moving at test speed reads as **broken**, not fast. The gating runner i tuned for signal and should stay that way — slow the demo script down on its own, and leave reading pauses where a viewer would actually need to read. +Two numbers, both from films that were re-cut for being too fast: + +- **Hold every screen for `max(caption read time, screen read time)`, floor 2.5s.** + Caption read time is roughly `words ÷ 3.5` seconds — which is what `narrate.js` + computes. Screen read time is how long it takes a viewer to *find the thing that + changed*, and it is always longer than it feels while authoring. `narrate.js` + knows only the caption; when the screen is the slower of the two, pass `holdMs`. +- **About two events per ten seconds.** A click, then its result. Not a click, a + scroll, a filter and a result — that is four things a viewer is asked to track + in the time they can follow one. + ### Give the compositor something to draw during pauses Playwright's video captures only frames the compositor actually produces. A @@ -154,6 +184,84 @@ matters of taste: picture no matter how good the synthesis is. Confirm the recorded file's duration matches the script's wall-clock before adding audio at all. +## The take has to be true, not only watchable + +`narrate.js` makes a recording watchable. Nothing in it — by design — checks that +what you filmed actually happened, or that the timestamp you cut on points at it. +Four failures, each of which cost a take and each of which *looked like success*: + +### The recorder's clock is not the video's clock + +It is wrong in **two** ways at once, and fixing only the first is the trap: + +- an **offset** — recording starts when the browser context is created, before + your first navigation has settled; +- a **scale** — the capture drops frames while the page is busy, so the file plays + back longer than the session it recorded. Measured at **~1.065**. + +A constant offset that was right at the start was **four seconds wrong by the +end** — the difference between cutting to the payoff screen and cutting to the one +before it. Three rounds of cuts showed the wrong moment in every beat before this +was found, and each one was plausible in isolation. + +So: record both anchors and map linearly, `video_t = A + B × mark_t`. Then +**verify by looking** — one frame from the middle of every clip, tiled into a +contact sheet. Spot-checking two clips is exactly how the wrong offset survived +those three rounds. + +### Assert the state the beat is about + +A click can be swallowed while the previous action's request is still in flight, +and the result looks fine: a board ended up *full but not solved*, so the payoff +never arrived and the control that depended on it stayed disabled. Check the DOM +for the state the beat is **about** — not that the click returned. + +A beat that cannot be asserted is a beat you cannot trust. This is not a verdict +about the app: the demo still never gates the build. It is a check on the +**recording**, and it belongs here for the same reason a camera has a viewfinder. + +### Pace to the app, not to the script + +Driving entries faster than the runtime committed them made two microflows overlap +and deadlock in Postgres. The `UpdateConflictException` surfaced as a modal dialog +that then swallowed every later click and killed the take. Two defences: never act +faster than a floor found experimentally per app, and detect-and-dismiss the error +dialog so one failure does not cost the session. + +Test the dialog guard on **visibility, not presence** — Mendix ships the error +dialog container in the DOM hidden, so a presence test fires on every click. + +### `recordVideo.size` pads; it does not scale + +A viewport smaller than the video size lands in the top-left corner with grey +around it. For a fixed-width page — which most Mendix layouts are — set the +viewport **to** the video size and apply CSS `zoom`: the page then lays out at the +smaller effective width while Chromium rasterizes at full device resolution. +Sharp and full-frame, where a smaller viewport is soft and letterboxed. A +stylesheet does not survive a navigation, so re-apply it after every `goto`. + +### These ship as code: `take.js` and `cut-clips.js` + +Both sit beside this file and are copied into the project with it. CommonJS, like +`narrate.js`, and required the same way. + +| | | +|---|---| +| `openTake(browser, opts)` | a context with `recordVideo`, both clock anchors, the zoom fix | +| `take.mark(name)` | a beat, timed from the settled first screen | +| `take.click` / `take.type` | paced to `minGap` and guarded against the error dialog | +| `take.assertBeat(name, probe, why)` | records whether the beat held; `finish()` **throws** if one did not | +| `take.finish()` | closes the context, writes `capture/beats.json` with `offset_s` | +| `node cut-clips.js` | cuts the raw take on the linear map, refuses implausible anchors, writes the contact sheet | + +`cut-clips.js` reads a project-owned `capture/clips.json` edit list, so the script +is the same everywhere and only the edit is per-film. It **refuses** a clip +shorter than its target unless that clip is explicitly marked `"freeze": true` — +holding a final frame is legitimate on a static screen, never to stretch an +interaction, and every pad is reported. + +What stays per-project is the walk and the selectors. Only the machinery is shared. + ## What to narrate Narrate only what a viewer with **no build context** would understand. @@ -164,9 +272,27 @@ Cut: and to a viewer it is noise about a claim they were not disputing - anything implying *"this used to be broken"* — the audience did not see the before, so it lands as an apology for a bug they never met +- **every word of Mendix vocabulary.** No entity, microflow, page, association, + domain model, catalog. The test is sharper than a word list: *if the visual + needs those words to make sense, the visual is wrong.* Keep it to the persona's own motivation: what they are trying to do, what they -see, and what changed for them. +see, and what changed for them. Name them — "Sam", not "the user". + +**Unless the product is single-player.** A puzzle, a calculator, a personal tool +has no task-persona, and inventing one is affectation. Write about the thing in +present tense instead. + +## Where this stops + +This skill owns the **capture**. How a capture is framed, cut and scored into a +finished film is the video system's — `video-system/` in `ako/mxcli-intro-video`, +which defines the product-demonstration type this skill feeds. + +Two boundaries worth keeping: the recording is **full-bleed** (no browser chrome, +no window frame, no laptop mockup — the capture *is* the frame), and the +narration plate stays `narrate.js`'s. **One caption system per film**; a second +one layered on in the edit reads as two designs. ## Checklist @@ -178,4 +304,11 @@ see, and what changed for them. - [ ] Recorded at **both** a desktop viewport and a real mobile device profile - [ ] The mobile pass runs the same steps, with nothing simplified - [ ] Narration mentions no database proof and no past bugs +- [ ] No Mendix vocabulary anywhere in the narration +- [ ] Every list the camera sees is populated with plausible data; every column + and button the camera sees is captioned +- [ ] Every beat that has a payoff is asserted, and the take reported none failed +- [ ] Both clock anchors recorded, and the contact sheet **looked at** — each tile + shows its own beat +- [ ] Every frozen tail is on a static screen, and reported - [ ] App was live and the database real — this instrument mocks nothing diff --git a/.claude/skills/mendix/record-narrated-demo/cut-clips.js b/.claude/skills/mendix/record-narrated-demo/cut-clips.js new file mode 100644 index 0000000000..7b16e390a2 --- /dev/null +++ b/.claude/skills/mendix/record-narrated-demo/cut-clips.js @@ -0,0 +1,160 @@ +// +// Cut a raw take into per-beat clips. +// +// node cut-clips.js [capture/beats.json] [capture/clips.json] +// +// The whole file exists because of one measured fact: THE RECORDER'S CLOCK IS +// NOT THE VIDEO'S CLOCK, and it is wrong in two ways at once. +// +// * an OFFSET — recording starts when the browser context is created, before +// your first navigation has settled (take.js records this as `offset_s`); +// * a SCALE — the capture drops frames while the page is busy, so the file +// plays back LONGER than the wall-clock session it recorded. Measured at +// ~1.065 on videos/sudoku-demo. +// +// Correcting only the offset is the trap: a constant that was right at the start +// was four seconds wrong by the end — the difference between cutting to the +// payoff screen and cutting to the screen before it. Three rounds of cuts showed +// the wrong moment in every beat before this was found, and each one looked +// plausible in isolation. So: two anchors, a linear map, and a contact sheet at +// the end because spot-checking two clips is exactly how it survived those three +// rounds. +// +// clips.json is the project's own edit list: +// +// { "clips": [ +// { "id": "03-home", +// "from": { "mark": "home", "offset": -1.20 }, +// "to": { "mark": "deal:click", "offset": -0.15 }, +// "target": 7.20, +// "freeze": true } +// ] } +// +// `target` is the finished clip length (voice duration + reading tail). `freeze` +// permits padding a SHORT clip by holding its final frame — legitimate on a +// static page, never to stretch an interaction, and always reported. +// + +const fs = require('fs'); +const path = require('path'); +const { execFileSync } = require('child_process'); + +const FFMPEG = process.env.HYPERFRAMES_FFMPEG_PATH || 'ffmpeg'; +const FFPROBE = process.env.HYPERFRAMES_FFPROBE_PATH || 'ffprobe'; + +const beatsFile = process.argv[2] || 'capture/beats.json'; +const clipsFile = process.argv[3] || 'capture/clips.json'; +const OUT = process.env.CLIPS_OUT || 'assets/clips'; + +// ffprobe missing is SILENT in more than one pipeline — it degrades rather than +// stopping. Check both binaries up front and treat a miss as fatal. +for (const bin of [FFMPEG, FFPROBE]) { + try { + execFileSync(bin, ['-version'], { stdio: 'ignore' }); + } catch { + console.error(`FATAL: ${bin} not on PATH. Set HYPERFRAMES_FFMPEG_PATH / HYPERFRAMES_FFPROBE_PATH.`); + process.exit(1); + } +} + +const meta = JSON.parse(fs.readFileSync(beatsFile, 'utf8')); +const spec = JSON.parse(fs.readFileSync(clipsFile, 'utf8')); +const src = path.join(path.dirname(beatsFile), 'raw', meta.video); +if (!fs.existsSync(src)) { + console.error(`FATAL: raw take not found: ${src}`); + process.exit(1); +} + +const duration = (file) => + Number(execFileSync(FFPROBE, ['-v', 'error', '-show_entries', 'format=duration', + '-of', 'csv=p=0', file]).toString().trim()); + +const rawDur = duration(src); +const markTime = (name) => { + const b = meta.beats.find((x) => x.name === name); + if (!b) throw new Error(`no such mark: '${name}' (have: ${meta.beats.map((x) => x.name).join(', ')})`); + return b.t; +}; +const lastMark = Math.max(...meta.beats.map((b) => b.t)); + +// The two anchors. A is where mark 0 sits in the file; B stretches wall-clock +// marks onto file time. +const A = typeof meta.offset_s === 'number' ? meta.offset_s : 0; +const B = (rawDur - A) / (lastMark + (meta.post_roll_s ?? 1.0)); + +console.log(` raw take ${rawDur.toFixed(2)}s, marks span ${lastMark.toFixed(2)}s`); +console.log(` video_t = ${A.toFixed(2)} + ${B.toFixed(4)} x mark_t`); + +// Reject rather than produce a plausible-looking wrong cut. +if (A < 0 || A > 30) throw new Error(`implausible start offset ${A}s — check the take`); +if (B < 0.95 || B > 1.25) throw new Error(`implausible clock scale ${B.toFixed(4)} — check the take`); + +const at = (point) => A + B * (markTime(point.mark) + (point.offset || 0)); + +fs.mkdirSync(OUT, { recursive: true }); +const report = []; + +for (const c of spec.clips) { + const from = at(c.from); + const avail = at(c.to) - from; + if (avail <= 0) throw new Error(`${c.id}: '${c.to.mark}' is not after '${c.from.mark}'`); + + const target = c.target ?? avail; + const short = target - avail; + if (short > 0.05 && !c.freeze) { + throw new Error( + `${c.id}: only ${avail.toFixed(2)}s of take for a ${target.toFixed(2)}s clip. ` + + `Set "freeze": true ONLY if this beat ends on a static screen; otherwise re-record it longer.`); + } + + const filters = ['fps=30']; + if (c.trimBottom) filters.unshift(`crop=in_w:in_h-${c.trimBottom}:0:0`); + // tpad holds the final frame; it is a no-op when the clip is long enough. + if (short > 0.05) filters.unshift(`tpad=stop_mode=clone:stop_duration=${short.toFixed(3)}`); + + const dst = path.join(OUT, `${c.id}.mp4`); + execFileSync(FFMPEG, [ + '-y', '-loglevel', 'error', '-ss', from.toFixed(3), '-i', src, + '-vf', filters.join(','), '-t', target.toFixed(3), + '-an', '-c:v', 'libx264', '-preset', 'slow', '-crf', '18', + '-pix_fmt', 'yuv420p', '-movflags', '+faststart', dst, + ], { stdio: ['ignore', 'inherit', 'inherit'] }); + + const out = duration(dst); + report.push({ + id: c.id, from: +from.toFixed(2), source_s: +avail.toFixed(2), + frozen_s: short > 0.05 ? +short.toFixed(2) : 0, out_s: +out.toFixed(2), + }); + console.log(` ${c.id.padEnd(16)} src ${from.toFixed(2)}s +${avail.toFixed(2)}s` + + `${short > 0.05 ? ` (+${short.toFixed(2)}s frozen)` : ''} -> ${out.toFixed(2)}s`); +} + +// Verify BY LOOKING. One frame from the middle of every clip, tiled — the check +// that would have caught the wrong offset on the first round instead of the +// fourth. Look at it before assembling: each tile must show its own beat. +const sheetDir = path.join(path.dirname(beatsFile), 'sheet'); +fs.rmSync(sheetDir, { recursive: true, force: true }); +fs.mkdirSync(sheetDir, { recursive: true }); +for (const r of report) { + execFileSync(FFMPEG, ['-y', '-loglevel', 'error', + '-ss', (r.out_s / 2).toFixed(2), '-i', path.join(OUT, `${r.id}.mp4`), + '-frames:v', '1', '-vf', 'scale=480:-1', path.join(sheetDir, `${r.id}.jpg`)], + { stdio: ['ignore', 'inherit', 'inherit'] }); +} +const sheet = path.join(path.dirname(beatsFile), 'contact-sheet.jpg'); +// Fit the grid to the clip count. A fixed tile size leaves most of the sheet +// black for a short film, which makes the thing you are supposed to study +// harder to read. +const cols = Math.min(3, report.length); +const rows = Math.ceil(report.length / cols); +execFileSync(FFMPEG, ['-y', '-loglevel', 'error', '-pattern_type', 'glob', + '-i', path.join(sheetDir, '*.jpg'), '-vf', `tile=${cols}x${rows}`, '-frames:v', '1', sheet], + { stdio: ['ignore', 'inherit', 'inherit'] }); + +fs.writeFileSync(path.join(path.dirname(beatsFile), 'clips-report.json'), + JSON.stringify({ source: meta.video, offset_s: A, clock_scale: +B.toFixed(4), clips: report }, null, 1)); + +console.log(`\n ${report.length} clips -> ${OUT}`); +const frozen = report.filter((r) => r.frozen_s); +if (frozen.length) console.log(` frozen tails: ${frozen.map((r) => `${r.id} +${r.frozen_s}s`).join(', ')}`); +console.log(` contact sheet -> ${sheet} <- LOOK AT THIS before assembling`); diff --git a/.claude/skills/mendix/record-narrated-demo/take.js b/.claude/skills/mendix/record-narrated-demo/take.js new file mode 100644 index 0000000000..589b9b118c --- /dev/null +++ b/.claude/skills/mendix/record-narrated-demo/take.js @@ -0,0 +1,197 @@ +// +// Recording integrity for a narrated demo. The companion to narrate.js: +// that file makes a take watchable, this one makes it TRUE — that the beat you +// meant to film actually happened, and that the timestamp you cut on points at +// it. +// +// Every guard here cost a take on ako/mxcli-intro-video (videos/sudoku-demo). +// None of it is theoretical, and none of it is a verdict about the app: a demo +// still never gates the build. These checks are about the RECORDING. +// +// Usage, from the per-project walkthrough script: +// +// const { openTake } = require('./take.js'); +// const take = await openTake(browser, { url: 'http://127.0.0.1:8080/', zoom: 1.68 }); +// await take.goto(); // navigates, settles, starts the clock +// take.mark('home'); +// await take.click('.sd-key >> nth=0'); // paced, dialog-guarded +// take.mark('entered'); +// await take.assertBeat('entered', () => page.locator('.sd-bad').count().then(n => n > 0), +// 'the conflict must be flagged, or beat 07 shows nothing'); +// await take.finish(); // closes context, writes beats.json +// + +const fs = require('fs'); +const path = require('path'); + +const DEFAULTS = { + url: 'http://127.0.0.1:8080/', + videoDir: 'capture/raw', + beatsFile: 'capture/beats.json', + // recordVideo.size PADS a smaller viewport into the video canvas — it does not + // scale it — so a 1140x641 viewport recorded at 1920x1080 puts the app in the + // top-left corner with grey around it. Keep viewport === size and reach the + // layout you want with `zoom` instead (below). + size: { width: 1920, height: 1080 }, + viewport: null, + // CSS zoom keeps the pixels native: at zoom 1.6842 a 1920px viewport lays out + // as 1140 CSS px (the width a fixed-width Mendix page wants) while Chromium + // still rasterizes at full device resolution. Sharp and full-frame, where + // shrinking the viewport is soft and letterboxed. + zoom: 1, + // Never drive the app faster than its runtime commits. Entering values back to + // back made two Mendix microflows overlap and deadlock in Postgres; the + // UpdateConflictException surfaced as a modal dialog that then swallowed every + // later click and killed the take. Find the floor experimentally per app. + minGap: 450, + dialogSelector: '.mx-dialog-error', + postRoll: 1.0, + // finish() throws when an asserted beat did not hold. The pipeline's default + // failure mode is to keep going and hand you a plausible-looking film with a + // dead beat in it; fail loudly instead. + strict: true, +}; + +async function openTake(browser, opts = {}) { + const o = { ...DEFAULTS, ...opts }; + o.viewport = o.viewport || o.size; + fs.mkdirSync(o.videoDir, { recursive: true }); + fs.mkdirSync(path.dirname(o.beatsFile), { recursive: true }); + + // ANCHOR 1 of 2. Playwright starts recording when the context is created — + // before your first navigation has even been issued, let alone settled. The + // gap between this instant and the first mark is the offset the cut must + // subtract. Capture it before newContext so it is never an underestimate. + const videoT0 = Date.now(); + const context = await browser.newContext({ + viewport: o.viewport, + recordVideo: { dir: o.videoDir, size: o.size }, + }); + const page = await context.newPage(); + + const beats = []; + const assertions = []; + let dialogs = 0; + let t0 = 0; + let lastAction = 0; + + const mark = (name) => { + const t = t0 ? (Date.now() - t0) / 1000 : 0; + beats.push({ name, t: Number(t.toFixed(2)) }); + console.log(` [${t.toFixed(2)}s] ${name}`); + return t; + }; + + // A stylesheet does not survive a navigation, so this is re-applied after + // every goto rather than set once. + const applyZoom = async (z) => { + const zoom = z || o.zoom; + if (zoom && zoom !== 1) await page.addStyleTag({ content: `html{zoom:${zoom};}` }); + }; + + // One runtime error must not cost the whole session: dismiss the dialog and + // carry on, but count it — a take with dialogs in it needs looking at. + const clearDialog = async () => { + const dlg = page.locator(o.dialogSelector).first(); + // Visibility, not presence. Mendix ships the error-dialog container in the + // DOM hidden, so a `count()` test fires on every single click: 800ms of + // dead time each time and a dialog count that is pure noise. + if (!(await dlg.isVisible().catch(() => false))) return false; + dialogs++; + console.log(` !! runtime error dialog (${dialogs}) — dismissing`); + const ok = dlg.locator('button').last(); + if (await ok.count()) await ok.click({ timeout: 5000 }).catch(() => {}); + await page.waitForTimeout(800); + return true; + }; + + const pace = async () => { + const since = Date.now() - lastAction; + if (lastAction && since < o.minGap) await page.waitForTimeout(o.minGap - since); + lastAction = Date.now(); + }; + + // Paced + dialog-guarded wrappers. Use these rather than page.click directly: + // the pacing is what keeps the runtime out of conflict, and the guard is what + // stops one dialog from eating every later action. + const click = async (selector, options) => { + await pace(); + await clearDialog(); + await page.locator(selector).first().click({ timeout: 15000, ...options }); + }; + const type = async (selector, text, options) => { + await pace(); + await clearDialog(); + await page.locator(selector).first().fill(text, { timeout: 15000, ...options }); + }; + + // Check the state the beat is ABOUT, not that a click returned. A click can be + // swallowed while the previous action's request is still in flight, and the + // result looks like success: the sudoku board twice ended up full but not + // solved, so the payoff never arrived and the control depending on it stayed + // disabled. A beat that cannot be asserted is a beat you cannot trust. + // + // `probe` returns truthy for "the beat happened". It is given the page. + const assertBeat = async (name, probe, why) => { + let ok = false; + let error = null; + try { + ok = !!(await probe(page)); + } catch (e) { + error = e.message; + } + assertions.push({ beat: name, ok, why, error }); + if (ok) { + console.log(` ok beat '${name}'`); + } else { + console.log(` !! beat '${name}' DID NOT HOLD — ${why}${error ? ` (${error})` : ''}`); + } + return ok; + }; + + const goto = async (url) => { + await page.goto(url || o.url, { waitUntil: 'networkidle', timeout: 90000 }); + await applyZoom(); + await page.waitForTimeout(1500); + // ANCHOR 2 of 2. The clock starts only once the first screen has settled, so + // every mark is measured from a frame a viewer would recognise. + t0 = Date.now(); + lastAction = 0; + }; + + const finish = async () => { + mark('end'); + await page.waitForTimeout(o.postRoll * 1000); + await context.close(); // flushes the video file + + const video = fs.readdirSync(o.videoDir).filter((f) => f.endsWith('.webm')).sort().pop(); + const failed = assertions.filter((a) => !a.ok); + const meta = { + video, + viewport: o.viewport, + video_size: o.size, + zoom: o.zoom, + dialogs, + post_roll_s: o.postRoll, + // The number the cut cannot be correct without. + offset_s: Number(((t0 - videoT0) / 1000).toFixed(3)), + beats, + assertions, + }; + fs.writeFileSync(o.beatsFile, JSON.stringify(meta, null, 1)); + + console.log(`\n video: ${path.join(o.videoDir, video || '(none)')}`); + console.log(` beats: ${beats.length} marks -> ${o.beatsFile}`); + console.log(` offset: ${meta.offset_s}s dialogs: ${dialogs}`); + if (failed.length) { + console.log(`\n ${failed.length} BEAT(S) DID NOT HOLD — this take is not usable as filmed:`); + for (const f of failed) console.log(` - ${f.beat}: ${f.why}`); + if (o.strict) throw new Error(`${failed.length} beat(s) did not hold; re-record rather than cutting this take`); + } + return meta; + }; + + return { page, context, mark, assertBeat, goto, applyZoom, clearDialog, click, type, pace, finish }; +} + +module.exports = { openTake, DEFAULTS }; diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e7c97413d..8444149325 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,32 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +- **`mxcli check` now catches two widget mistakes that only `exec` caught** — an explicit widget id that resolves to nothing (**MDL-WIDGET25**) and a real container keyword used on a parent that declares no such container (**MDL-WIDGET26**, e.g. `group` inside HTML Element). Both parsed cleanly and reported success at check time, then failed the build. + + The cause is that the **grammar was the widget-kind validator**: `widgetTypeV3` is an allow-list, so an unknown kind could not parse and the validator never needed an independent notion of one — and neither of these mistakes is a keyword the parser checks. `isUniversalObjectListKeyword` actively suppressed the second, treating a container keyword as always-an-item wherever it appeared. + + Both rules stay silent when they cannot be sure, and each guard is load-bearing. **MDL-WIDGET25 needs a project**: with none, the registry holds only mxcli's nine embedded widgets and every real project widget would be called unknown. With one, an id whose `.mpk` **is** installed is real and merely unextracted — the registry `check` uses reads `.mxcli/widgets/` and does not refresh from installed packages. **MDL-WIDGET26 needs a resolvable parent**, since a parent mxcli cannot see declares no containers as far as it knows. Measured: zero false positives across the whole `mdl-examples/doctype-tests/` corpus with a project, and none without. + +- **`DESCRIBE WIDGET` — a widget's definition, in-language** — `describe widget combobox;` or `describe widget 'com.mendix.widget.web.htmlelement.HTMLElement';` reports each property's key, type, caption, category, required flag, default and enumeration values, plus the dynamic rules the widget's editor uses to *hide* properties under some configurations (the ones that cause CE0463 when written into the pruned half). + + A widget was the only MDL extension point without one: a microflow, nanoflow, Java action and JavaScript action all describe in-language against the live project. That gap is *why* `mxcli widget init` generates markdown documentation at all — and why that documentation could drift from what the parser accepts, as reported in mendixlabs/mxcli#1036. The statement and `mxcli widget describe` are now the same function, so they cannot disagree. + + It also reports the widget's **body containers** — child slots and object lists, with each object list's item properties — and marks which are **authorable from MDL today**, since most are not (30 of 46 across a stock project). That answer is derived by parsing a probe, never from a list: the defect behind #1036 was two keyword lists with nothing comparing them, and a third list here would repeat it one layer up. It also means the marks correct themselves when the grammar catches up. + + It emits an **MDL example that parses as written**. The head form (`gallery widget1` vs `pluggablewidget '' widget1`) and every container in it are chosen by probing the real parser, and whatever the grammar cannot yet express is left out *and named* — including required properties that need a real entity or microflow from your project. This is the half of the generated `.md` that was wrong: its example failed on its own first line. Because both halves are derived rather than written down, the example widens on its own as the grammar gains ground. + + Bindings the example cannot fill in — datasource, attribute, action, expression, selection — are **named rather than invented**, and narrowed by the widget's own visibility rules: a property is required only where the editor shows it, so Combo box's eleven drop to six under its default configuration (and would drop further; `16 of 32 editor hide-rules recognized` is the current ceiling). Pruning is conservative by design — an indeterminable condition never prunes, and a rule about an object-list item never prunes the widget's own property. + + It works with **no project open**, answering from mxcli's embedded set — "what can I write here?" is asked before anything is open. With a project the answer is better: the installed `.mpk` is version-accurate and is the only place a Marketplace widget appears. + +- **A widget with no definition no longer names a remedy that cannot work** — `exec` failed with `no definition for widget … (run 'mxcli widget init -p app.mpr')`, and running that command changed nothing, because `widget init` scans `widgets/` and the package was not there. Reported as the postscript to mendixlabs/mxcli#1036, where it cost a debugging session. The message now branches on whether the package is actually installed, using the same `FindMPK` lookup the template loader makes before giving up, and otherwise says to install the widget. + + Measured while fixing it: a widget whose package is absent is one **Studio Pro cannot use either**, so mxcli ships no definitions for these. A blank Mendix 11.13 project carries 33 widgets and none of File Uploader, Events, Google Tag or Markdown viewer; installing File Uploader takes `widgets/` from 33 to 34, and a page using it then builds with no `widget init` at all — `initPluggableEngine` refreshes definitions from installed packages on its own. + +- **`()` is accepted on every widget** — `container c ()`, `dynamictext t ()` and `pluggablewidget 'id' pw ()` were parse errors reported at the closing paren, as though the widget were wrong, while bare `container c` and `container c (x: 'y')` both parsed. `widgetPropertiesV3` required at least one property; an empty list is now allowed, removing an arbitrary difference between two spellings of the same thing. + +- **Generated widget docs emit child slots with their required name** — `mxcli widget init` wrote `tagcontentcontainer { … }`, which even a working slot rejects, so the three child slots that *did* parse were documented in a form that could not. Names are emitted and numbered (`slot1`, `slot2`), since two identically named widgets on one page would collide. + - **A failed build now says which test caused it** (ako/mxcli-sudoku FINDINGS #46 follow-up) — an `@expect` that is syntactically valid but only rejected by MxBuild took down an entire `mxcli test --local` run: no test results at all, valid tests in the same file never executed, and the cause arrived as ~200 lines of mxbuild JSON with the real error among dozens of unrelated Atlas warnings. `BuildResult` parsed only the status and message and left the rest of the response unread, though mxbuild returns every problem with a severity, an error code and a location. Measured on 11.13, a failing build returns **18 problems of which one is the error**, so printing the body meant 11,580 bytes in which nothing marked the line that mattered. Filtering to errors renders it as `[CE0117] Error(s) in expression. — at MxTest / Microflow 'Test_test_3' / Decision '$result = 3'`. diff --git a/cmd/mxcli/cmd_check.go b/cmd/mxcli/cmd_check.go index 7bc5f91791..4864bd5075 100644 --- a/cmd/mxcli/cmd_check.go +++ b/cmd/mxcli/cmd_check.go @@ -23,6 +23,23 @@ Pass -p and it also resolves every reference — modules, entities, pages, microflows and icons — against that project; --references is implied by -p and is kept only for compatibility. +It also resolves MEMBER names, not just the entity they belong to: an attribute +named in a create/change activity is looked up on that entity and its +generalizations, so a typo is reported here rather than as CE1613 at the far end +of a build. This needs the target's entity to be known, which it is for a create +(the entity is in the statement) and for a change on a parameter, a database +retrieve, an association retrieve, or a loop over one of those. A variable bound +by some other activity is left unchecked rather than guessed at. + +The same applies inside widgets: a page's XPath constraint has every step +resolved against the entity it filters, and a template parameter (ContentParams +/ CaptionParams) rooted in a variable is reported with no project at all. + +Expression kinds are checked where the position declares one: a bare word as a +create/change member's value (Mendix expressions have no bare identifiers), and +a log message's template parameter, which must be a String — Mendix does not +coerce there, so wrap a non-String one in toString(...). + Reference validation is smart: it automatically skips references to objects that are created within the script itself. For example, if your script creates a module "MyModule" and then creates entities in it, no error will be reported diff --git a/cmd/mxcli/cmd_widget_describe.go b/cmd/mxcli/cmd_widget_describe.go index 76912fc597..dcd9a13923 100644 --- a/cmd/mxcli/cmd_widget_describe.go +++ b/cmd/mxcli/cmd_widget_describe.go @@ -4,15 +4,9 @@ package main import ( "encoding/json" - "fmt" - "path/filepath" - "sort" "strings" "github.com/mendixlabs/mxcli/mdl/executor" - "github.com/mendixlabs/mxcli/mdl/types" - mwidgets "github.com/mendixlabs/mxcli/modelsdk/widgets" - mmpk "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" "github.com/spf13/cobra" ) @@ -24,6 +18,14 @@ properties (key, type, caption, category, required, default, enum options) and t dynamic property rules (which properties the widget's editor hides under which configuration) lifted from the widget package's editorConfig. +It also reports the widget's BODY CONTAINERS — its object lists (a repeated entry, +e.g. a chart series) and child slots (a block of widgets, e.g. a gallery template) +— with whether MDL can express each one, and ends with a complete MDL example +that parses and checks as written. Values in the example are real enumeration +members; bindings it cannot fill (a datasource, an attribute, an action) are named +under "omitted" rather than invented, because a generic example cannot know a name +from your project. + The widget can be named by its MDL keyword (e.g. COMBOBOX, DATAGRID2) or its full widget id (e.g. com.mendix.widget.web.combobox.Combobox). @@ -45,93 +47,18 @@ func init() { } // describedProperty is one property of a widget's discovered format. -type describedProperty struct { - Key string `json:"key"` - Type string `json:"type"` - Caption string `json:"caption,omitempty"` - Category string `json:"category,omitempty"` - Required bool `json:"required"` - Default string `json:"default,omitempty"` - System bool `json:"system,omitempty"` - Enum []string `json:"enum,omitempty"` - Children []describedProperty `json:"children,omitempty"` -} - -// describedRule is one dynamic (visibility) rule of a widget's discovered format. -type describedRule struct { - Property string `json:"property"` - HiddenWhen string `json:"hiddenWhen"` -} - -// widgetDescription is the full inspection result (also the JSON shape). -type widgetDescription struct { - WidgetID string `json:"widgetId"` - MDLName string `json:"mdlName,omitempty"` - Name string `json:"name,omitempty"` - Version string `json:"version,omitempty"` - Source string `json:"source"` // "project .mpk" | "embedded template" - Kind string `json:"kind,omitempty"` - Properties []describedProperty `json:"properties"` - Rules []describedRule `json:"dynamicRules"` - RuleCoverage string `json:"ruleCoverage,omitempty"` -} - +// The description itself is built by executor.DescribeWidget, which the MDL +// statement `DESCRIBE WIDGET x` also calls. One code path on purpose: a widget +// was the only MDL extension point with no in-language DESCRIBE, and the +// generated documentation that filled the gap could drift from what the parser +// accepts (mendixlabs/mxcli#1036). Two implementations would reopen that. func runWidgetDescribe(cmd *cobra.Command, args []string) error { - arg := args[0] projectPath, _ := cmd.Flags().GetString("project") format, _ := cmd.Flags().GetString("format") - registry, err := executor.NewWidgetRegistry() + desc, err := executor.DescribeWidget(args[0], projectPath) if err != nil { - return fmt.Errorf("failed to create widget registry: %w", err) - } - if projectPath != "" { - _ = registry.LoadUserDefinitions(projectPath) - } - - // Resolve the target widget id + optional built-in definition. - widgetID, def := resolveWidgetTarget(registry, arg) - if widgetID == "" { - return widgetNotFoundError(registry, arg) - } - - desc := widgetDescription{WidgetID: widgetID} - if def != nil { - desc.MDLName = def.MDLName - desc.Kind = def.WidgetKind - } - if desc.Kind == "" { - desc.Kind = "pluggable" - } - - // Properties + version: prefer the project's installed .mpk (version-accurate, - // includes marketplace widgets); else fall back to mxcli's embedded template. - if projectPath != "" { - if dir := projectDirOf(projectPath); dir != "" { - if mpkPath, ferr := mmpk.FindMPK(dir, widgetID); ferr == nil && mpkPath != "" { - if wd, perr := mmpk.ParseMPKForWidget(mpkPath, widgetID); perr == nil && wd != nil { - desc.Name = wd.Name - desc.Version = wd.Version - desc.Source = "project .mpk" - desc.Properties = propsFromMPK(wd) - desc.Rules, desc.RuleCoverage = rulesFromProject(mpkPath, widgetID) - } - } - } - } - if desc.Source == "" { - // Embedded template fallback. - tmpl, terr := mwidgets.GetTemplate(widgetID) - if terr != nil || tmpl == nil { - return fmt.Errorf("no installed .mpk and no embedded template for %q — try -p to inspect a project widget", arg) - } - desc.Name = tmpl.Name - desc.Version = tmpl.Version - desc.Source = "embedded template" - desc.Properties = propsFromTemplate(tmpl.Type) - if def != nil { - desc.Rules = rulesFromDef(def.PropertyVisibility) - } + return err } if strings.EqualFold(format, "json") { @@ -139,277 +66,6 @@ func runWidgetDescribe(cmd *cobra.Command, args []string) error { enc.SetIndent("", " ") return enc.Encode(desc) } - printWidgetDescription(cmd, desc) + executor.PrintWidgetDescription(cmd.OutOrStdout(), *desc) return nil } - -// resolveWidgetTarget maps a CLI argument (MDL keyword or widget id) to a widget id -// and, when known, the built-in WidgetDefinition. A dotted argument is treated as a -// widget id directly. -func resolveWidgetTarget(registry *executor.WidgetRegistry, arg string) (string, *executor.WidgetDefinition) { - if strings.Contains(arg, ".") { - if def, ok := registry.GetByWidgetID(arg); ok { - return arg, def - } - return arg, nil // unknown to the registry, but a valid id to look up in the project - } - upper := strings.ToUpper(arg) - if def, ok := registry.Get(upper); ok { - return def.WidgetID, def - } - // Well-known widgets that are special-cased in the executor (no .def.json in the - // registry) but that users still name by keyword. - if id, ok := builtinWidgetAliases[upper]; ok { - def, _ := registry.GetByWidgetID(id) - return id, def - } - return "", nil -} - -// builtinWidgetAliases maps MDL keywords for executor-special-cased widgets (which -// have no .def.json registry entry) to their widget ids, so `widget describe` can -// resolve them by the same friendly names users write in MDL. -var builtinWidgetAliases = map[string]string{ - "DATAGRID": "com.mendix.widget.web.datagrid.Datagrid", - "DATAGRID2": "com.mendix.widget.web.datagrid.Datagrid", -} - -// widgetNotFoundError builds a helpful error listing the known MDL names. -func widgetNotFoundError(registry *executor.WidgetRegistry, arg string) error { - var names []string - for _, d := range registry.All() { - if d.MDLName != "" { - names = append(names, d.MDLName) - } - } - for alias := range builtinWidgetAliases { - names = append(names, strings.ToLower(alias)) - } - sort.Strings(names) - return fmt.Errorf("unknown widget %q — use an MDL keyword (%s) or a full widget id (com.mendix.widget…). Run `mxcli widget list` to see all", - arg, strings.Join(names, ", ")) -} - -// projectDirOf returns the directory containing widgets/ for a project path -// (accepts either the .mpr file or its directory). -func projectDirOf(projectPath string) string { - if strings.EqualFold(filepath.Ext(projectPath), ".mpr") { - return filepath.Dir(projectPath) - } - return projectPath -} - -// propsFromMPK builds described properties from a parsed .mpk definition, in the -// widget's declared order (regular + system interleaved). -func propsFromMPK(wd *mmpk.WidgetDefinition) []describedProperty { - order := wd.AllTopLevel - if len(order) == 0 { - order = wd.Properties - } - out := make([]describedProperty, 0, len(order)) - for _, p := range order { - out = append(out, describedPropFromMPK(p)) - } - return out -} - -func describedPropFromMPK(p mmpk.PropertyDef) describedProperty { - dp := describedProperty{ - Key: p.Key, - Type: p.Type, - Caption: p.Caption, - Category: p.Category, - Required: p.Required, - Default: p.DefaultValue, - System: p.IsSystem, - } - if dp.System && dp.Type == "" { - dp.Type = "system" - } - for _, ev := range p.EnumValues { - dp.Enum = append(dp.Enum, ev.Key) - } - for _, c := range p.Children { - dp.Children = append(dp.Children, describedPropFromMPK(c)) - } - return dp -} - -// propsFromTemplate walks an embedded template's Type map (ObjectType.PropertyTypes) -// to build described properties. Used when no project .mpk is available. -func propsFromTemplate(typ map[string]any) []describedProperty { - objType, _ := typ["ObjectType"].(map[string]any) - pts, _ := objType["PropertyTypes"].([]any) - var out []describedProperty - for _, pt := range pts { - m, ok := pt.(map[string]any) - if !ok { - continue // leading array marker - } - out = append(out, describedPropFromTemplate(m)) - } - return out -} - -func describedPropFromTemplate(m map[string]any) describedProperty { - dp := describedProperty{ - Key: asString(m["PropertyKey"]), - Caption: asString(m["Caption"]), - Category: asString(m["Category"]), - } - vt, _ := m["ValueType"].(map[string]any) - if vt != nil { - dp.Type = asString(vt["Type"]) - dp.Default = asString(vt["DefaultValue"]) - if r, ok := vt["Required"].(bool); ok { - dp.Required = r - } - if evs, ok := vt["EnumerationValues"].([]any); ok { - for _, ev := range evs { - if em, ok := ev.(map[string]any); ok { - if k := asString(em["_Key"]); k != "" { - dp.Enum = append(dp.Enum, k) - } - } - } - } - if nested, ok := vt["ObjectType"].(map[string]any); ok { - if npts, ok := nested["PropertyTypes"].([]any); ok { - for _, npt := range npts { - if nm, ok := npt.(map[string]any); ok { - dp.Children = append(dp.Children, describedPropFromTemplate(nm)) - } - } - } - } - } - dp.System = isSystemPropKey(dp.Key) - return dp -} - -func isSystemPropKey(key string) bool { - switch key { - case "Label", "Visibility", "Editability", "Name", "TabIndex": - return true - } - return false -} - -// rulesFromProject extracts dynamic rules from the project's installed .mpk editor -// config, returning the rules and a coverage note (recognized / total hide-calls). -func rulesFromProject(mpkPath, widgetID string) ([]describedRule, string) { - rules, recognized, total := executor.ExtractWidgetVisibilityStats(mpkPath, widgetID) - coverage := "" - if total > 0 { - coverage = fmt.Sprintf("%d of %d editor hide-rules recognized", recognized, total) - } - return rulesToDescribed(rules), coverage -} - -func rulesFromDef(rules []types.WidgetVisibilityRule) []describedRule { - return rulesToDescribed(rules) -} - -func rulesToDescribed(rules []types.WidgetVisibilityRule) []describedRule { - out := make([]describedRule, 0, len(rules)) - for _, r := range rules { - if r.HiddenWhen == nil { - continue - } - out = append(out, describedRule{Property: r.PropertyKey, HiddenWhen: conditionText(r.HiddenWhen)}) - } - sort.Slice(out, func(i, j int) bool { return out[i].Property < out[j].Property }) - return out -} - -// conditionText renders a visibility condition as readable English. -func conditionText(c *types.WidgetVisibilityCondition) string { - switch c.Operator { - case "eq": - return fmt.Sprintf("%s = %q", c.PropertyKey, c.Value) - case "ne": - return fmt.Sprintf("%s ≠ %q", c.PropertyKey, c.Value) - case "truthy": - return fmt.Sprintf("%s is set", c.PropertyKey) - case "falsy": - return fmt.Sprintf("%s is not set", c.PropertyKey) - default: - return fmt.Sprintf("%s %s %q", c.PropertyKey, c.Operator, c.Value) - } -} - -func asString(v any) string { - s, _ := v.(string) - return s -} - -func printWidgetDescription(cmd *cobra.Command, d widgetDescription) { - out := cmd.OutOrStdout() - title := d.Name - if title == "" { - title = d.WidgetID - } - fmt.Fprintf(out, "Widget: %s", title) - if d.MDLName != "" { - fmt.Fprintf(out, " (%s)", d.MDLName) - } - fmt.Fprintln(out) - fmt.Fprintf(out, " ID: %s\n", d.WidgetID) - if d.Version != "" { - fmt.Fprintf(out, " Version: %s\n", d.Version) - } - fmt.Fprintf(out, " Kind: %s\n", d.Kind) - fmt.Fprintf(out, " Source: %s\n", d.Source) - - fmt.Fprintf(out, "\nProperties (%d):\n", countProps(d.Properties)) - printProps(out, d.Properties, 0) - - fmt.Fprintf(out, "\nDynamic property rules (%d):\n", len(d.Rules)) - if len(d.Rules) == 0 { - fmt.Fprintln(out, " (none discovered)") - } - for _, r := range d.Rules { - fmt.Fprintf(out, " %-40s hidden when %s\n", r.Property, r.HiddenWhen) - } - if d.RuleCoverage != "" { - fmt.Fprintf(out, " — %s\n", d.RuleCoverage) - } -} - -func countProps(props []describedProperty) int { - n := 0 - for _, p := range props { - n++ - n += countProps(p.Children) - } - return n -} - -func printProps(out interface{ Write([]byte) (int, error) }, props []describedProperty, depth int) { - indent := strings.Repeat(" ", depth+1) - for _, p := range props { - req := "" - if p.Required { - req = " required" - } - sys := "" - if p.System { - sys = " [system]" - } - line := fmt.Sprintf("%s%-34s %-13s", indent, p.Key, p.Type) - extra := strings.TrimRight(req+sys, " ") - if p.Default != "" { - extra = strings.TrimSpace(extra + " default=" + p.Default) - } - if len(p.Enum) > 0 { - extra = strings.TrimSpace(extra + " {" + strings.Join(p.Enum, "|") + "}") - } - if p.Category != "" { - extra = strings.TrimSpace(extra + " (" + p.Category + ")") - } - fmt.Fprintf(out, "%s %s\n", strings.TrimRight(line, " "), extra) - if len(p.Children) > 0 { - printProps(out, p.Children, depth+1) - } - } -} diff --git a/cmd/mxcli/cmd_widget_describe_test.go b/cmd/mxcli/cmd_widget_describe_test.go index c7b3cfa173..7d3c1fff70 100644 --- a/cmd/mxcli/cmd_widget_describe_test.go +++ b/cmd/mxcli/cmd_widget_describe_test.go @@ -8,7 +8,6 @@ import ( "testing" "github.com/mendixlabs/mxcli/mdl/executor" - "github.com/mendixlabs/mxcli/mdl/types" ) // TestWidgetDescribe_EmbeddedCombobox runs `widget describe COMBOBOX --format json` @@ -28,7 +27,7 @@ func TestWidgetDescribe_EmbeddedCombobox(t *testing.T) { if err := runWidgetDescribe(cmd, []string{"COMBOBOX"}); err != nil { t.Fatalf("describe COMBOBOX json: %v", err) } - var d widgetDescription + var d executor.WidgetDescription if err := json.Unmarshal([]byte(out.String()), &d); err != nil { t.Fatalf("unmarshal json: %v\n%s", err, out.String()) } @@ -59,35 +58,3 @@ func TestWidgetDescribe_EmbeddedCombobox(t *testing.T) { } // TestWidgetDescribe_UnknownWidget reports a helpful error. -func TestWidgetDescribe_UnknownWidget(t *testing.T) { - reg, err := executor.NewWidgetRegistry() - if err != nil { - t.Fatalf("registry: %v", err) - } - id, _ := resolveWidgetTarget(reg, "NOPE") - if id != "" { - t.Errorf("resolveWidgetTarget(NOPE) = %q, want empty", id) - } - // DATAGRID2 resolves via the builtin alias even without a .def.json entry. - if id, _ := resolveWidgetTarget(reg, "datagrid2"); id != "com.mendix.widget.web.datagrid.Datagrid" { - t.Errorf("resolveWidgetTarget(datagrid2) = %q", id) - } -} - -// TestConditionText renders the four operators as readable English. -func TestConditionText(t *testing.T) { - cases := []struct { - op, val, want string - }{ - {"eq", "None", `itemSelection = "None"`}, - {"ne", "Multi", `itemSelection ≠ "Multi"`}, - {"truthy", "", "itemSelection is set"}, - {"falsy", "", "itemSelection is not set"}, - } - for _, c := range cases { - got := conditionText(&types.WidgetVisibilityCondition{PropertyKey: "itemSelection", Operator: c.op, Value: c.val}) - if got != c.want { - t.Errorf("op %s: got %q, want %q", c.op, got, c.want) - } - } -} diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index b729006912..fc7bbb899c 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -277,6 +277,7 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "ATTRIBUTES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "FILTERTYPE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "IMAGE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "GLYPH", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "QUEUE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "QUEUES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "SCHEDULED", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, diff --git a/cmd/mxcli/syntax/features_domain_model.go b/cmd/mxcli/syntax/features_domain_model.go index f9df6bf952..b96ed8a90e 100644 --- a/cmd/mxcli/syntax/features_domain_model.go +++ b/cmd/mxcli/syntax/features_domain_model.go @@ -136,7 +136,7 @@ func init() { "many-to-one", "many-to-many", "foreign key", "owner", "delete behavior", }, - Syntax: "[@anchor(from: (x, y), to: (x, y))]\nCREATE [OR MODIFY] ASSOCIATION Module.Name\n FROM Module.FromEntity TO Module.ToEntity\n TYPE Reference|ReferenceSet\n [OWNER Default|Both]\n [DELETE_BEHAVIOR behavior]\n [COMMENT 'text'];\nALTER ASSOCIATION Module.Name SET ANCHOR FROM (x, y) TO (x, y);\nDROP ASSOCIATION Module.Name;\n\nOR MODIFY: updates type/owner/delete behavior in-place, preserves UUID.\n\n@anchor sets the LINE ANCHORS — where the connector attaches to each entity box\nin the domain model editor — as a PERCENTAGE of the box (0..100, whole numbers).\n`from` is the FROM entity's box, `to` the TO entity's: (0, 50) is the middle of\nthe left edge, (100, 50) the right, (50, 100) the bottom centre. Omitting an end\nPRESERVES what is stored, so a CREATE OR MODIFY about something else never\nflattens a hand-tuned line. Cross-module associations have no anchors.", + Syntax: "[@anchor(from: (x, y), to: (x, y))]\nCREATE [OR MODIFY] ASSOCIATION Module.Name\n FROM Module.FromEntity TO Module.ToEntity\n TYPE Reference|ReferenceSet\n [OWNER Default|Both]\n [DELETE_BEHAVIOR behavior]\n [COMMENT 'text'];\nALTER ASSOCIATION Module.Name SET ANCHOR FROM (x, y) TO (x, y);\nDROP ASSOCIATION Module.Name;\n\nOR MODIFY: updates type/owner/delete behavior in-place, preserves UUID.\n\n@anchor sets the LINE ANCHORS — where the connector attaches to each entity box\nin the domain model editor — as a PERCENTAGE of the box (0..100, whole numbers).\n`from` is the FROM entity's box, `to` the TO entity's: (0, 50) is the middle of\nthe left edge, (100, 50) the right, (50, 100) the bottom centre. Omitting an end\nPRESERVES what is stored, so a CREATE OR MODIFY about something else never\nflattens a hand-tuned line. Cross-module associations have no anchors.\n\nDROP reconciles the entity access rules that named the association, and is\nREFUSED while a message definition still exposes it (that one cannot be\nreconciled -- removing the member would change a published contract). The\nrefusal prints the `alter message definition ... drop member` statement for\neach definition, ready to run.", Example: "-- Many-to-one\nCREATE ASSOCIATION Shop.Order_Customer\n FROM Shop.Order TO Shop.Customer\n TYPE Reference\n OWNER Default\n DELETE_BEHAVIOR DELETE_BUT_KEEP_REFERENCES;\n\n-- Many-to-many\nCREATE ASSOCIATION Shop.Product_Tag\n FROM Shop.Product TO Shop.Tag\n TYPE ReferenceSet\n OWNER Both;\n\n-- Line leaving the bottom of Order and entering the top of Customer\n@anchor(from: (50, 100), to: (50, 0))\nCREATE ASSOCIATION Shop.Order_Customer\n FROM Shop.Order TO Shop.Customer;\n\n-- Retune the line without restating the association\nALTER ASSOCIATION Shop.Order_Customer SET ANCHOR FROM (0, 54) TO (100, 54);", SeeAlso: []string{"domain-model.association.create", "domain-model.association.anchor", "domain-model.association.delete-behavior"}, }) @@ -241,7 +241,7 @@ func init() { "caption", "show enumerations", "describe enumeration", "drop enumeration", }, - Syntax: "CREATE ENUMERATION Module.Name (\n ValueName 'Display Caption',\n ...\n);\n\nALTER ENUMERATION Module.Name ADD VALUE NewValue [CAPTION 'Display Caption'];\nALTER ENUMERATION Module.Name RENAME VALUE OldName TO NewName;\nALTER ENUMERATION Module.Name MODIFY VALUE ValueName CAPTION 'New Caption';\nALTER ENUMERATION Module.Name DROP VALUE ValueName;\n\nSHOW ENUMERATIONS;\nSHOW ENUMERATIONS IN ;\nDESCRIBE ENUMERATION Module.Name;\nDROP ENUMERATION Module.Name;\n\nUsing in entity:\n AttrName: Enumeration(Module.EnumName)", + Syntax: "CREATE ENUMERATION Module.Name (\n ValueName 'Display Caption',\n ...\n);\n\nALTER ENUMERATION Module.Name ADD VALUE [IF NOT EXISTS] NewValue [CAPTION 'Display Caption'];\nALTER ENUMERATION Module.Name RENAME VALUE OldName TO NewName;\nALTER ENUMERATION Module.Name MODIFY VALUE ValueName CAPTION 'New Caption';\nALTER ENUMERATION Module.Name DROP VALUE [IF EXISTS] ValueName;\n\nIF NOT EXISTS / IF EXISTS make a script RE-RUNNABLE. Without them the second\nrun errors and exec STOPS THERE, so one already-present value leaves every\nlater statement unapplied. A defensive drop-then-add is not a substitute: the\ndrop fails when the value is absent and the add when it is present.\n\nSHOW ENUMERATIONS;\nSHOW ENUMERATIONS IN ;\nDESCRIBE ENUMERATION Module.Name;\nDROP ENUMERATION Module.Name;\n\nUsing in entity:\n AttrName: Enumeration(Module.EnumName)", Example: "CREATE ENUMERATION MyModule.OrderStatus (\n Pending 'Pending Approval',\n Processing 'Being Processed',\n Shipped 'Shipped to Customer'\n);\n\n-- Using in an entity\nCREATE PERSISTENT ENTITY MyModule.Order (\n OrderNumber: String(20) NOT NULL,\n Status: Enumeration(MyModule.OrderStatus)\n);", SeeAlso: []string{"domain-model.enumeration", "domain-model.entity.attributes"}, }) diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 9522aea0c3..7ab1fdbebd 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -29,6 +29,83 @@ func init() { SeeAlso: []string{"page", "page.widgets", "page.datasource"}, }) + Register(SyntaxFeature{ + Path: "page.widget-describe", + Summary: "DESCRIBE WIDGET — a widget's properties, enum values and editor rules", + Keywords: []string{ + "describe widget", "widget properties", "widget definition", "what properties", + "enum values", "widget rules", "hidden properties", "pluggable widget properties", + }, + Syntax: `DESCRIBE WIDGET ; +DESCRIBE WIDGET '';`, + Example: `DESCRIBE WIDGET combobox; +DESCRIBE WIDGET 'com.mendix.widget.web.htmlelement.HTMLElement'; + +-- Names the widget by its MDL keyword or its full widget id. +-- +-- Works with NO project open, answering from mxcli's embedded set. With a +-- project the answer is better: the installed .mpk is version-accurate and is +-- the only place a Marketplace widget appears. +-- +-- Reports each property's key, type, caption, category, whether it is required, +-- its default and its enumeration values, plus the dynamic rules the widget's +-- editor uses to HIDE properties under some configurations — the ones that +-- cause CE0463 if written into the pruned half. +-- +-- Also emits an MDL example that PARSES AS WRITTEN: the head form and every +-- container in it are chosen by probing the real parser, and anything the +-- grammar cannot yet express is left out and named. So the example widens on +-- its own as MDL gains ground, and cannot promise syntax that fails. +-- +-- Same output as ` + "`mxcli widget describe`" + `, because it is the same code. + +-- The other direction — which pages already use it — is a reference query, +-- and needs ` + "`refresh catalog full`" + `: +SHOW REFERENCES TO combobox; +SHOW IMPACT OF htmlelement; + +-- Name it as you write it in a page body; the casing does not matter. A +-- built-in Mendix widget (textbox, dynamictext) has no definition and so no +-- reference edge — use SHOW WIDGETS for those.`, + SeeAlso: []string{"page.widgets", "page.create"}, + }) + + Register(SyntaxFeature{ + Path: "page.widget-any", + Summary: "Any widget with a definition, written by its own MDL name", + Keywords: []string{ + "htmlelement", "html element", "fileuploader", "file uploader", "markdown", + "custom widget syntax", "marketplace widget", "pluggable widget name", + "widget not recognized", "mismatched input", "def-driven", "mdl name", + "object list", "child slot", "widget container", "attributes list", + }, + Syntax: ` [( Prop: Value, ... )] [{ }] + [( Prop: Value, ... )] [{ ... }]`, + Example: `-- Any widget with a definition is written by its own MDL name. There is no +-- list of blessed keywords: if ` + "`describe widget `" + ` knows it, you can write it. +CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { + htmlelement frame (tagName: 'div') { + -- object lists and child slots the widget's own definition declares + attribute a1 (attributeName: 'title', attributeValueType: 'expression') + tagcontentcontainer body { + dynamictext t (Content: 'hello') + } + } + fileuploader up () +} + +-- The names come from the widget itself, so ask it rather than guessing: +-- mxcli widget describe htmlelement -p app.mpr +-- which lists every property, every container, and an example that parses. +-- +-- A name that resolves to no definition is MDL-WIDGET25 (widget) or +-- MDL-WIDGET26 (container), each naming the near misses — but BOTH need a +-- project, because the set of valid widget names IS the project's installed +-- packages. With no -p, ` + "`mxcli check`" + ` cannot tell a typo from a widget it +-- has simply never seen, and says nothing rather than guessing.`, + SeeAlso: []string{"page.widgets", "page.widget-describe", "page.create"}, + }) + Register(SyntaxFeature{ Path: "page.widgets", Summary: "Widget types: containers, data widgets, inputs, actions, display", diff --git a/cmd/mxcli/syntax/widget_keywords_drift_test.go b/cmd/mxcli/syntax/widget_keywords_drift_test.go index 1e3808d9de..622b902ed6 100644 --- a/cmd/mxcli/syntax/widget_keywords_drift_test.go +++ b/cmd/mxcli/syntax/widget_keywords_drift_test.go @@ -60,7 +60,20 @@ func widgetTypeAlternatives(t *testing.T) []string { line = line[:i] } for _, tok := range strings.Split(line, "|") { - if tok = strings.TrimSpace(tok); regexp.MustCompile(`^[A-Z][A-Z0-9]*$`).MatchString(tok) { + tok = strings.TrimSpace(tok) + // IDENTIFIER is the generic alternative added by slice 2 of + // PROPOSAL_def_driven_widget_bodies.md, not a keyword — it is how a + // widget is named by its own MDL name. It lexes as a token class, so + // the all-caps shape below matches it; there is no keyword + // "identifier" to document. Its lower-case sibling `keyword` (slice + // 3) is already skipped by that shape. + // + // Covered instead by TestDefDrivenWidgetNameIsDocumented below, which + // is the assertion that actually applies to it. + if tok == "IDENTIFIER" { + continue + } + if regexp.MustCompile(`^[A-Z][A-Z0-9]*$`).MatchString(tok) { out = append(out, strings.ToLower(tok)) } } @@ -164,3 +177,45 @@ func TestWidgetKeywordGuardCanFail(t *testing.T) { "whole guard is vacuous") } } + +// The enumerated keywords are no longer the whole widget vocabulary: since +// slices 2-3 any widget with a definition can be named by its MDL name, and any +// container a definition declares can be written in a body. That is a bigger +// capability than the list above, and the same reasoning applies to it — a +// capability nobody can find is one people build around. +// +// This asserts both halves are still present in the grammar (so their removal +// is noticed) and that a page.* topic tells the reader about them. +func TestDefDrivenWidgetNameIsDocumented(t *testing.T) { + path := filepath.Join("..", "..", "..", "mdl", "grammar", "domains", "MDLPage.g4") + b, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + rule := regexp.MustCompile(`(?s)\nwidgetTypeV3\s*\n\s*:(.*?)\n\s*;`).FindStringSubmatch(string(b)) + if rule == nil { + t.Fatal("widgetTypeV3 rule not found") + } + body := rule[1] + for _, alt := range []string{"IDENTIFIER", "keyword"} { + if !regexp.MustCompile(`\|\s*` + alt + `\s*\n`).MatchString(body + "\n") { + t.Errorf("widgetTypeV3 no longer offers the generic `%s` alternative — "+ + "if that was deliberate, 30 of 46 documented widget containers stop parsing again "+ + "(mendixlabs/mxcli#1036); update this guard rather than deleting it", alt) + } + } + + corpus := pageSyntaxCorpus() + if corpus == "" { + t.Fatal("no page.* topics registered — the guard would pass vacuously") + } + // The reader has to be able to learn that a widget can be named by its own + // name. Any of these phrasings satisfies that. + for _, want := range []string{"mdl name", "def-driven", "any widget with a definition"} { + if strings.Contains(corpus, want) { + return + } + } + t.Error("no page.* syntax topic explains that a widget can be written by its own MDL name " + + "(e.g. `htmlelement frame (...)`) — add it to cmd/mxcli/syntax/features_page.go") +} diff --git a/docs-site/src/appendixes/error-messages.md b/docs-site/src/appendixes/error-messages.md index 6c844a7ed8..3aa32df789 100644 --- a/docs-site/src/appendixes/error-messages.md +++ b/docs-site/src/appendixes/error-messages.md @@ -78,7 +78,7 @@ page MyModule.OrderList: widget `cb1` (combobox) has no property **Cause:** The property key written on a pluggable widget is not declared in the widget's `.def.json` (the extracted schema from its `.mpk`). Usually a typo; sometimes a property that exists in a different widget but not this one. **Solution:** -1. Compare the key against the widget's known properties — `mxcli describe widget ` lists them. +1. Compare the key against the widget's known properties — `mxcli widget describe ` lists them (or `describe widget ;` in MDL). 2. Use the suggested replacement if one is offered (Levenshtein-nearest match). 3. If the property genuinely doesn't exist on this widget version, check that `.mxcli/widgets/` has the latest schema: `mxcli refresh catalog -p app.mpr` re-extracts any `.mpk` whose mtime changed. 4. If the property was just added by a `.mpk` upgrade, make sure `mxcli init` or `widget init` was run after the upgrade. diff --git a/docs-site/src/internals/catalog-schema.md b/docs-site/src/internals/catalog-schema.md index 01e3177e26..30fa0eed95 100644 --- a/docs-site/src/internals/catalog-schema.md +++ b/docs-site/src/internals/catalog-schema.md @@ -156,19 +156,68 @@ CREATE TABLE WIDGETS ( ### REFS +The reference graph: one row per edge. Populated by `refresh catalog full`. + ```sql CREATE TABLE REFS ( - SourceName TEXT, -- Referencing document - SourceKind TEXT, -- "Microflow", "Page", etc. - TargetName TEXT, -- Referenced element - TargetKind TEXT, -- "Entity", "Microflow", etc. - RefKind TEXT -- "Call", "DataSource", "Association", etc. + Id INTEGER PRIMARY KEY AUTOINCREMENT, + SourceType TEXT NOT NULL, -- "MICROFLOW", "PAGE", "ENTITY", ... + SourceId TEXT NOT NULL, -- element $ID, or '' where the builder has no id + SourceName TEXT NOT NULL, -- referencing document, module-qualified + TargetType TEXT NOT NULL, -- "ENTITY", "MICROFLOW", "WIDGET", ... + TargetId TEXT, -- element $ID, or the widget ID for a WIDGET target + TargetName TEXT NOT NULL, -- referenced element + RefKind TEXT NOT NULL, -- see the vocabulary below + ModuleName TEXT, + ProjectId TEXT, + SnapshotId TEXT ); -CREATE INDEX idx_refs_source ON REFS(SourceName); -CREATE INDEX idx_refs_target ON REFS(TargetName); +CREATE INDEX idx_refs_source ON refs(SourceType, SourceName); +CREATE INDEX idx_refs_target ON refs(TargetType, TargetName); +CREATE INDEX idx_refs_kind ON refs(RefKind); ``` +`RefKind` values are lower-case, and the current vocabulary is whatever +`CATALOG.GRAPH_REFKIND_DISTRIBUTION` reports for your project — query that +rather than trusting a list here: + +| RefKind | Edge | +|---------|------| +| `call` | flow calls a microflow / nanoflow / rule / Java action / REST operation | +| `create` / `change` / `delete` / `retrieve` | flow acts on an entity object | +| `return` | flow returns an entity type | +| `parameter` | page or flow parameter entity type | +| `generalize` | entity extends entity | +| `associate` | association targets entity | +| `layout` | page uses a layout | +| `datasource` | page or widget reads an entity | +| `action` | widget calls a microflow / nanoflow | +| `show_page` | flow or widget action opens a page | +| `home_page` / `login_page` / `menu_item` | navigation profile references a page | +| `calculate` | calculated attribute uses a microflow | +| `schedule` | scheduled event runs a microflow | +| `validate` | attribute validation rule uses a regular expression | +| `widget` | page or snippet uses a pluggable / custom widget | + +#### WIDGET targets + +A `widget` edge is the odd one out and is worth knowing about before you join +against it: + +- `TargetName` is the widget's **MDL name** (`COMBOBOX`), not its dotted widget + ID. The ID is in `TargetId`. A dotted target would be mis-read as a module by + `GRAPH_MODULE_COUPLING` and friends, which take everything before the first + dot as the module name. +- It is therefore the only `TargetName` that is not module-qualified — a widget + definition belongs to no Mendix module. `GRAPH_GOD_NODES` excludes `WIDGET` + targets from its asset list for that reason, while still counting a page's + out-degree towards the widgets it uses. +- Only widgets with a definition get an edge. A built-in Mendix widget + (`Forms$DynamicText`) has none, so it produces no row; use `CATALOG.WIDGETS` + for those. +- One edge per page x widget, not per widget instance. + ### PERMISSIONS ```sql @@ -234,6 +283,16 @@ WHERE AttributeCount > 20 ORDER BY AttributeCount DESC; SELECT SourceName, RefKind FROM CATALOG.REFS WHERE TargetName = 'Sales.Customer'; +-- Which pages use a given pluggable widget? +SELECT SourceType, SourceName FROM CATALOG.REFS +WHERE RefKind = 'widget' AND TargetName = 'COMBOBOX'; + +-- Which installed widget packages does nothing use? +-- (MDL's SELECT has no NOT EXISTS / NOT IN — use an anti-join.) +SELECT d.MdlName, d.WidgetId FROM CATALOG.WIDGET_DEFINITIONS d +LEFT JOIN CATALOG.REFS r ON r.TargetId = d.WidgetId AND r.RefKind = 'widget' +WHERE r.Id IS NULL; + -- Full-text search SELECT name, kind, snippet(STRINGS, 2, '', '', '...', 20) FROM CATALOG.STRINGS WHERE strings MATCH 'validation error'; diff --git a/docs-site/src/language/widget-types.md b/docs-site/src/language/widget-types.md index 72ed451bf5..ed29b93255 100644 --- a/docs-site/src/language/widget-types.md +++ b/docs-site/src/language/widget-types.md @@ -1,11 +1,13 @@ # Widget Types -MDL supports a comprehensive set of widget types for building Mendix pages. Each widget is declared with a type keyword, a unique name, properties in parentheses, and optional child widgets in braces. +A widget is declared with a type keyword, a unique name, properties in parentheses, and optional child widgets in braces. ```sql WIDGET_TYPE widgetName (Property: value, ...) [{ children }] ``` +**The list below is not the boundary.** The built-in Mendix widgets are documented here because they have fixed, hand-written property sets. Every **pluggable or custom widget** installed in the project is also written by its own name, with a body derived from the widget's definition — see [Any installed widget](#any-installed-widget) below. If a widget is in `widgets/`, MDL can name it. + ## Widget Categories | Category | Widgets | @@ -445,6 +447,74 @@ NAVIGATIONLIST navMain { } ``` +## Any installed widget + +Everything above is a **built-in** widget: its keyword and properties are fixed +by Mendix and by mxcli. A **pluggable or custom widget** — DataGrid 2, Combo box, +Gallery, HTML Element, the charts, anything from the Marketplace, anything your +team built — is named the same way, by its own MDL name: + +```sql +htmlelement frame (tagName: 'div', tagContentMode: 'container') { + attribute a1 (attributeName: 'data-testid', attributeValueType: 'expression') + tagcontentcontainer body { + dynamictext caption (Content: 'Inside the element') + } +} +``` + +Three things there come from the widget's own definition rather than from any +list in mxcli: + +- **The keyword** `htmlelement` — the widget's MDL name, which is the last + segment of its widget id. +- **The properties** `tagName`, `tagContentMode` — written with the widget's own + spelling, exactly as `DESCRIBE WIDGET` reports them. +- **The body containers** `attribute` (an object list, one entry per repetition) + and `tagcontentcontainer` (a child slot, holding widgets). + +### Finding the names + +Ask the widget: + +```sql +DESCRIBE WIDGET htmlelement; +``` + +It lists every property with its type, default and enumeration members, every +body container and whether MDL can express it, and a complete example that +parses and checks as written. See +[DESCRIBE WIDGET](../reference/query/describe-widget.md). + +### The explicit form + +A widget can also be named by its full id. This is the fallback, not the norm — +use it when two installed packages ship the same MDL name, or when you have the +id in hand and not the name: + +```sql +pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' frame ( + tagName: 'div' +) +``` + +`DESCRIBE PAGE` emits the short form wherever it round-trips, and falls back to +the id form when the name would be ambiguous. + +### When a widget is not found + +A name that resolves to no installed definition is an error, not a silently +accepted widget — MDL-WIDGET25, with the nearest known names suggested. A +container keyword the parent widget does not declare is MDL-WIDGET26. Both need +a project open (`-p`), since without one mxcli knows only its embedded widgets. + +If a widget you have installed is not found, its definition has not been +extracted yet: + +```bash +mxcli widget init -p app.mpr # extract definitions for every widget in widgets/ +``` + ## Common Widget Properties These properties are shared across many widget types: @@ -464,3 +534,5 @@ These properties are shared across many widget types: - [Page Structure](./page-structure.md) -- layout selection and data sources - [Data Binding](./data-binding.md) -- connecting widgets to attributes - [ALTER PAGE](./alter-page.md) -- modifying widgets in existing pages +- [DESCRIBE WIDGET](../reference/query/describe-widget.md) -- inspect any installed widget's properties and body containers +- [Pluggable Widgets Across Versions](../guides/pluggable-widgets.md) -- how mxcli keeps widget definitions version-correct diff --git a/docs-site/src/reference/integration/create-message-definition-collection.md b/docs-site/src/reference/integration/create-message-definition-collection.md index f548d1f218..934108c7a9 100644 --- a/docs-site/src/reference/integration/create-message-definition-collection.md +++ b/docs-site/src/reference/integration/create-message-definition-collection.md @@ -33,18 +33,23 @@ association with its own member list — the same discriminator import and expor mappings use. Naming the association's target entity is required, and not decoration. The -stored cardinality tracks the **direction of traversal**, not the association's -type: - -| traversal | cardinality | -|---|---| -| from the association's FROM entity (following the foreign key) | a single object | -| from its TO entity (the reverse) | a list | - -So the same association gives a single object one way and a list the other. An -association that connects the two entities in **neither** direction is refused — -a wrong cardinality builds cleanly and would silently expose a list as a single -object. +stored cardinality is derived from the **direction of traversal** and the +**association's type** together: + +| traversal | `Reference` | `ReferenceSet` | +|---|---|---| +| from the association's FROM entity (following the reference) | a single object | a list | +| from its TO entity (the reverse) | a list | a list | + +So a `Reference` gives a single object one way and a list the other, while a +`ReferenceSet` is a list in both directions — a set is many at both ends. + +An association that connects the two entities in **neither** direction is +refused rather than guessed at. The two halves of the rule fail differently if +you do get one wrong, which is useful when diagnosing: a wrong *direction* +builds cleanly and silently exposes a list as a single object, while a wrong +*type* is caught by mxbuild as CE6524 on the definition ("The occurrence of +'...' has changed") plus CE0295 on any mapping element bound to it. **Inherited attributes** are named exactly like the entity's own; mxcli resolves each to the entity that declares it, which is what Mendix stores. diff --git a/docs-site/src/reference/query/README.md b/docs-site/src/reference/query/README.md index 331dc23679..57815af5e9 100644 --- a/docs-site/src/reference/query/README.md +++ b/docs-site/src/reference/query/README.md @@ -27,6 +27,7 @@ Statements for browsing and inspecting project elements. Query statements are re | [DESCRIBE ENUMERATION](describe-enumeration.md) | Show enumeration values and documentation | | [DESCRIBE MICROFLOW](describe-microflow.md) | Show complete MDL source for a microflow or nanoflow | | [DESCRIBE PAGE](describe-page.md) | Show complete MDL source for a page or snippet | +| [DESCRIBE WIDGET](describe-widget.md) | Show a widget's properties, body containers and a working MDL example | ## Search diff --git a/docs-site/src/reference/query/describe-widget.md b/docs-site/src/reference/query/describe-widget.md new file mode 100644 index 0000000000..969d7be7c4 --- /dev/null +++ b/docs-site/src/reference/query/describe-widget.md @@ -0,0 +1,113 @@ +# DESCRIBE WIDGET + +## Synopsis + + DESCRIBE WIDGET + + DESCRIBE WIDGET '' + +## Description + +Shows the format mxcli has discovered for a pluggable or custom widget: its +properties (key, type, caption, category, required, default, enumeration +members), the **body containers** it accepts, the editor rules that hide a +property under some configurations, and a complete MDL example. + +A widget was the only MDL extension point without a `DESCRIBE`. That is why +`mxcli widget init` writes markdown documentation at all — and why the two could +drift. `DESCRIBE WIDGET` and `mxcli widget describe` are the same function, so +they cannot disagree. + +Unlike the other `DESCRIBE` statements, this one **works with no project open**: +"what can I write here?" is a question asked before anything is open. With `-p`, +the properties and rules come from the widget package actually installed in the +project (`widgets/*.mpk`) — version-accurate, and the only place a Marketplace +widget appears at all. Without it, they come from mxcli's embedded template. + +## Parameters + +*keyword* +: The widget's MDL name, as written in a page body — `combobox`, `htmlelement`, + `datagrid`. Case-insensitive. + +*widget id* +: The full widget id as a quoted string — + `'com.mendix.widget.web.htmlelement.HTMLElement'`. Use this form for a widget + whose MDL name is ambiguous, or when you have the id in hand from a `.mpk`. + +## Examples + +```sql +DESCRIBE WIDGET htmlelement; +``` + +Example output, abbreviated: + +``` +Widget: HTML Element (htmlelement) + ID: com.mendix.widget.web.htmlelement.HTMLElement + Version: 1.2.2 + Kind: pluggable + Source: project .mpk + +Properties (23): + tagName enumeration required default=div {div|span|p|ul|…} + tagContentMode enumeration required default=container {container|innerHTML} + attributes object (General::HTML attributes) + attributeName string required + attributeValueType enumeration required default=expression {expression|template} + +Body containers (4): + attribute object list -> attributes authorable + items: attributeName, attributeValueType, … + event object list -> events authorable + items: eventName, eventAction, … + tagcontentcontainer child slot -> tagContentContainer authorable + tagcontentrepeatcontainer child slot -> tagContentRepeatContainer authorable + +MDL example (parses as written): + htmlelement widget1 ( + tagName: 'div', + tagUseRepeat: false, + tagContentMode: 'container' + ) { + attribute item1 (attributeValueType: 'expression') -- one entry of `attributes` + event item2 (eventName: 'onClick') -- one entry of `events` + tagcontentcontainer slot3 { + -- widgets for `tagContentContainer` + } + } +``` + +By widget id: + +```sql +DESCRIBE WIDGET 'com.mendix.widget.web.htmlelement.HTMLElement'; +``` + +## Notes + +**Body containers report whether MDL can express them.** `authorable` is derived +by parsing a probe against the live grammar, never read from a list — so the mark +is correct by construction rather than by maintenance. A container reported as +not authorable is one to set in Studio Pro. + +**The MDL example parses and checks as written.** The whole block is parsed +before it is emitted, and its property values are real members rather than +placeholders. Bindings it cannot fill — a datasource, an attribute, an action — +are **named rather than invented**, under `-- omitted:`, because a generic +example cannot know a name from your project. + +**The example is narrowed by the widget's own editor rules.** A property the +widget hides under the configuration the example picked is left out, so what you +see is what that configuration actually supports. The footer reports how many of +the widget's hide-rules were recognised; an unrecognised rule never prunes. + +**`LIST WIDGETS` does not exist**, deliberately. `SHOW WIDGETS` already means +widget *instances placed on pages*, and the definitions are +`SELECT * FROM CATALOG.WIDGET_DEFINITIONS`. + +## See Also + +[SHOW WIDGETS](show-widgets.md), [DESCRIBE PAGE](describe-page.md), +[Pluggable Widgets Across Versions](../../guides/pluggable-widgets.md) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 9ffb5dff36..3889d34624 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -72,7 +72,7 @@ create persistent entity Module.Photo ( | Rename association | `rename association Module.Old to New;` | Updates all references | | Show entities | `show entities [in module];` | List all or filter by module | | Create enumeration | `create [or modify] enumeration Module.Name (Value1 'caption', ...);` | | -| Alter enumeration values | `alter enumeration Module.Name add value X [caption '..'] \| rename value X to Y \| modify value X caption '..' \| drop value X;` | `modify value … caption` re-captions in place (works while referenced) | +| Alter enumeration values | `alter enumeration Module.Name add value [if not exists] X [caption '..'] \| rename value X to Y \| modify value X caption '..' \| drop value [if exists] X;` | `modify value … caption` re-captions in place (works while referenced). `if not exists` / `if exists` make the script re-runnable — the bare forms error and stop the run | | Drop enumeration | `drop enumeration Module.Name;` | | | Create association | `create [or modify] association Module.Name from Parent to Child type reference\|ReferenceSet [owner default\|both] [delete_behavior ...];` | OR MODIFY updates existing association in-place. **The FROM entity must live in `Module`** — Mendix stores an association in its FROM entity's module, so a remote FROM writes a dangling pointer and the project stops OPENING (**MDL070**). The TO entity may be remote; that direction is stored BY NAME | | Drop association | `drop association Module.Name;` | | @@ -775,6 +775,14 @@ create or replace navigation Responsive ); ``` +**An item with no icon is reported (MDL074, a warning).** The navigation sidebar +collapses to an icon rail, and that is the state most users leave it in: a +collapsed item shows its icon, and one without falls back to the first few +characters of its caption — rarely enough to tell `Orders` from `Order lines`. +The menu still builds and `mx check` passes, so the only symptom is in a browser. +The rule covers every item at every depth, in both `create navigation`'s `menu` +block and `create menu`, and needs no project. + `icon` is optional and is a **qualified name** into an **icon collection** — `Atlas_Core.Atlas`, `Atlas_Core.Atlas_Filled`, `Atlas_Core.Atlas_Styling`, or one of your own — written like any other model reference. Hyphenated Atlas names @@ -782,10 +790,27 @@ of your own — written like any other model reference. Hyphenated Atlas names `Atlas_Core.Atlas."align-center"`. List the available names with `describe icon collection Atlas_Core.Atlas`. -Studio Pro can also set a *glyph* icon (a numeric character code) or an *image* -icon (pointing into an image collection). Those are different elements; MDL -writes only the icon-collection form, and `describe navigation` marks the other -two with a comment instead of emitting an `icon` clause that would convert them. +Mendix stores **three different icon elements**, and each has its own form, +because they are not spellings of one value — a collection icon and an image icon +each hold a qualified name (into an icon collection and an *image* collection, +different documents), while a glyph icon holds a numeric character code and no +name at all: + +| form | element | holds | +|------|---------|-------| +| `icon Atlas_Core.Atlas.home` | `Forms$IconCollectionIcon` | a name in an icon collection | +| `icon glyph 57377` | `Forms$GlyphIcon` | a numeric character code | +| `icon image MyModule.Images.logo` | `Forms$ImageIcon` | a name in an image collection | + +The bare form is the icon-collection icon, so every existing script keeps its +meaning. The keyword forms exist because writing a bare name for an image icon +would rebuild it as a collection icon — a silent variant swap. + +`describe navigation` emits all three, so describe → exec is lossless. It +previously wrote a comment for the other two, and since `create or replace +navigation` is a **full replacement**, re-running that output DELETED the icon +the comment had just declined to describe. A `$Type` this build does not know is +still flagged rather than guessed at. ## Project Settings @@ -1238,6 +1263,7 @@ MDL uses explicit property declarations for pages: | Pop-up dimensions | `PopupWidth: n, PopupHeight: n, PopupResizable: bool` | `(Layout: Atlas_Core.PopupLayout, PopupWidth: 800, PopupHeight: 480, PopupResizable: true)` — case-sensitive; default 600×600 | | Page CSS class / style | `Class: 'css-class', Style: 'css: rule'` | `(Title: 'Home', Class: 'container-fluid bg-light', Style: 'min-height: 100vh')` — the page's Appearance | | Page variables | `variables: { $name: type = 'expr' }` | `variables: { $show: boolean = 'true' }` | +| Inspect a widget | `describe widget ;` | `describe widget combobox;` — properties, enum values, defaults and the editor rules that HIDE properties under some configurations. Works with no project open; with one, reads the installed `.mpk` (version-accurate, and the only place a Marketplace widget appears). Same output as `mxcli widget describe` | | Widget name | Required after type | `textbox txtName (...)` | | Attribute binding | `attribute: AttrName` | `textbox txt (label: 'Name', attribute: Name)` | | Variable binding | `datasource: $Var` | `dataview dv (datasource: $Product) { ... }` | @@ -1617,6 +1643,15 @@ Cross-reference commands require `refresh catalog full` to populate reference da `show callers` covers invocation only. A document that merely *uses a type* — an entity as a page datasource, a microflow parameter, an entity's generalization — is not a caller of it; `show references to` lists those. +A **pluggable or custom widget** is a reference target too, so "which pages use this widget?" is one query — the same question about a Java action always was: + +```mdl +show references to combobox; -- pages and snippets that place a Combo box +show impact of htmlelement; -- the same, grouped by document type +``` + +Name the widget the way you write it in a page body. The target is stored as the widget's MDL name and matched case-insensitively when the exact spelling finds nothing, so `combobox`, `ComboBox` and `COMBOBOX` all resolve; the resolved spelling is printed. A built-in Mendix widget (`textbox`, `dynamictext`) has no definition and therefore no edge — use `show widgets` for those. + ## Connection & Session | Statement | Syntax | Notes | diff --git a/docs/11-proposals/PROPOSAL_authorable_message_definitions.md b/docs/11-proposals/PROPOSAL_authorable_message_definitions.md index d692bbb961..22c7eb80bd 100644 --- a/docs/11-proposals/PROPOSAL_authorable_message_definitions.md +++ b/docs/11-proposals/PROPOSAL_authorable_message_definitions.md @@ -127,6 +127,20 @@ wrong: get it backwards and the definition exposes a list as a single object, or a single object as a list. A build error is not guaranteed — the mapping over it simply carries the wrong cardinality. +> **Correction (ako/mxcli-rest FINDINGS #60).** The heading above is right that +> `MaxOccurs` is not a function of the type *alone*; it is a function of the +> direction **and** the type. The census could not see the second half: every +> association in it is a `Reference`, so the corpus never varied the input the +> rule was declared independent of. A **`ReferenceSet` is a list in both +> directions**, and unlike the direction half this one does have a build error +> behind it — mxbuild reports CE6524 on the definition and CE0295 on any mapping +> element bound to it. Shipped rule: +> +> | | forward (holder is FROM) | reverse | +> |---|---|---| +> | `Reference` | `1` | `-1` | +> | `ReferenceSet` | `-1` | `-1` | + **Design consequence: the statement names the target entity**, so direction is explicit in the source text rather than inferred: diff --git a/docs/11-proposals/PROPOSAL_def_driven_widget_bodies.md b/docs/11-proposals/PROPOSAL_def_driven_widget_bodies.md new file mode 100644 index 0000000000..02fa4d79ea --- /dev/null +++ b/docs/11-proposals/PROPOSAL_def_driven_widget_bodies.md @@ -0,0 +1,604 @@ +--- +title: Widgets as first-class MDL, not a second dialect +status: draft +date: 2026-09-04 +related: + - PROPOSAL_mcp_pluggable_widget_authoring.md + - PROPOSAL_multi_version_pluggable_widgets.md + - PROPOSAL_widget_property_visibility.md +--- + +# Proposal: Widgets as first-class MDL, not a second dialect + +**Status:** Draft +**Date:** 2026-09-04 + +Using a widget should feel like calling a Java action. It does not. A widget is +named differently, described differently, and is invisible to the reference +graph — and where every other extension point resolves against the project, a +widget resolves against a hardcoded list in the grammar. This proposal closes +the four gaps, in order of how much each one costs. + +## Problem Statement + +### What a user hit + +[mendixlabs/mxcli#1036](https://github.com/mendixlabs/mxcli/issues/1036). A team +needed a sandboxed iframe: the HTML Element widget's `attributes` object list +would carry `sandbox` and `srcdoc`. MDL could not express it, so they fell back +to `tagContentHTML`, **which executes same-origin — precisely the risk the +sandbox exists to prevent.** + +The generated documentation is what cost them the time. `widget init` writes +`.claude/skills/widgets/htmlelement.md`, whose lead example is: + +```sql +PLUGGABLEWIDGET 'com.mendix.widget.web.htmlelement.HTMLElement' widget1 { + tagcontentcontainer { ... } + attribute item1 -- one entry of `attributes` + event item1 -- one entry of `events` +} +``` + +Fed back into `mxcli check` verbatim, it fails on the first line of its own body: + +``` +line 4:4 mismatched input 'tagcontentcontainer' expecting '}' +``` + +The file is written *for LLM agents to follow* and carries the widget's real +property table, so it reads as authoritative. Multiple sessions concluded the +feature existed. + +### The real shape of the problem + +The reported bug is one symptom of a widget being a second dialect inside MDL. +Every other extension point — a submicroflow, a Java action, a JavaScript +action — is referenced by qualified name, described in-language, and findable in +the reference graph: + +```mdl +$R = CALL MICROFLOW Module.Name (Param = value) +$R = CALL NANOFLOW Module.Name (Param = value) +$R = CALL JAVA ACTION Module.Name (Param = value) +$R = CALL JAVASCRIPT ACTION Module.Name (Param = value) +``` + +A widget matches none of that: + +| | reference | `DESCRIBE` | "who uses it?" | +|---|---|---|---| +| microflow / nanoflow | `Module.Name` | MDL statement | `call` edge | +| java action | `Module.Name` | MDL statement, round-trips | `call` edge | +| javascript action | `Module.Name` | MDL statement | `call` edge | +| **widget** | keyword *if blessed*, else a string ID | **CLI command only** | **no edge at all** | + +Four gaps follow, and each is measured below. + +### Gap 1 — Two reference forms, gated on a hardcoded list + +``` +htmlelement.def.json: mdlName: HTMLELEMENT widgetId: com.mendix.widget.web.htmlelement.HTMLElement + +htmlelement h (tagName: 'div') -- REJECTED +combobox c (Attribute: Name) -- PARSES +``` + +A widget gets a keyword only if it appears in the grammar's `widgetTypeV3` list. +Everything else must be written by string ID. + +**The resolution machinery already exists and is already wired.** +`cmd_pages_builder_v3.go:425` tries the MDL name *first*: + +```go +// Try by MDL name first +if def, ok := pb.widgetRegistry.Get(strings.ToUpper(w.Type)); ok { + return pb.buildPluggable(def, w) +} +``` + +Every `.def.json` carries an `mdlName`; `WidgetRegistry.Get(mdlName)` exists. +The only thing missing is a parser that will produce `w.Type == "htmlelement"`. +Today `MDLName` is read **only to build error messages**. + +### Gap 2 — Object lists and child slots, the same defect one level down + +Inside a widget body, containers are gated on nine hardcoded keywords +(`MDLPage.g4:402–410`), while the doc generator derives one for **every** object +list and child slot mechanically (`deriveObjectListKeyword` singularises any +property key; child slots use `strings.ToUpper(child.Key)`). Two lists, nothing +comparing them. + +Measured against the fixture project (33 widget defs, 46 documented constructs): + +| | parses | rejected | +|---|---|---| +| Object lists | 13/16 | `attribute`, `event`, `attr` | +| Child slots (named, under `pluggablewidget`) | 3/30 | 27 | +| **Total** | **16/46** | **30 (65%)** | + +20 of 33 widgets document at least one keyword that cannot parse. **Control** — +identical widget, identical body shape, only list membership differs: + +```mdl +group g1 (headerText: 'x') -- PARSES +attribute a1 (attributeName: 'title') -- REJECTED +``` + +### Gap 3 — No `DESCRIBE WIDGET`, which is why the generated doc exists at all + +`DESCRIBE JAVA ACTION FeedbackModule.ValidateEmail` emits re-executable MDL. +There is no MDL equivalent for a widget — only `mxcli widget describe`, a CLI +command. + +This reframes the reported bug. Actions need no generated documentation because +`DESCRIBE` answers in-language, against the live project. **The widget `.md` is a +workaround for a missing statement**, and the reason it could drift is that +nothing else could answer the question. Fixing the generator alone treats the +symptom. + +### Gap 4 — Widgets are absent from the reference graph + +`CATALOG.REFS` carries 15 edge kinds and none is widget use: + +``` +action associate call change create datasource delete generalize +home_page layout menu_item parameter retrieve return show_page +``` + +So `show references to ` returns +`MICROFLOW FeedbackModule.VAL_Feedback | call`, and the same question about a +widget returns nothing. Impact analysis for a widget upgrade is unanswerable. +This is the same class as the scheduled-event gap recorded in CLAUDE.md, where a +microflow run only by a scheduled event read as dead until a `schedule` edge was +added. + +### Why honest documentation is not sufficient + +Option 2 of the issue (emit only what the grammar accepts) converts a silent +failure into a documented dead end. Worth doing, and it is slice 1 below — but +it would not have unblocked the reporter. There is no other route: `ALTER PAGE` +rejects the same construct, verified with a working control. + +```mdl +alter page M.Sandbox { + insert into frame { dynamictext t1 (Content: 'hello') } -- PARSES (control) +}; +alter page M.Sandbox { + insert into frame { attribute a1 (attributeName: 'sandbox') } -- REJECTED +}; +``` + +They would have had accurate documentation of a capability gap, and shipped the +same-origin fallback anyway. + +## BSON Structure + +**No new BSON and no new write path.** This is what makes the proposal small, +and it should be re-verified before code is written, because everything rests on +it. + +`PluggableWidgetEngine.applyObjectLists` (`widget_engine.go:1068`) is already +fully def-driven: + +```go +byContainer[strings.ToUpper(lists[i].MDLContainer)] = &lists[i] +... +mapping, ok := byContainer[strings.ToUpper(child.Type)] +``` + +It matches the AST child's `Type` **string** against whatever the def declares, +and knows nothing about the nine keywords. The visitor sets that string from the +token's literal text (`visitor_page_v3.go:554`): + +```go +widget.Type = strings.ToLower(typeCtx.GetText()) +``` + +So the whole pipeline below the parser is text-driven and generic. The keyword +lists exist **only so ANTLR has a token to match** — an artefact of the parser +generator leaking out as a capability boundary. + +`PLUGGABLEWIDGET` and `CUSTOMWIDGET` are additionally **already the same thing**: +both take the `buildPluggable` branch (`cmd_pages_builder_v3.go:430`) and both +store `CustomWidgets$CustomWidget`. + +## Proposed MDL Syntax + +### A widget is named like everything else + +```mdl +create page Sales.Frame ( Title: 'Frame', Layout: Atlas_Core.Atlas_Default ) +{ + htmlelement frame ( tagName: 'div' ) { + attribute sandboxAttr ( + attributeName: 'sandbox', + attributeValueType: 'template', + attributeValueTemplate: 'allow-scripts' + ) + event onClickEvent ( eventName: 'onClick' ) + tagcontentcontainer content { + dynamictext note ( Content: 'Sandboxed' ) + } + } +} +``` + +Nothing here is new syntax — it is the shape `combobox`, `datagrid` and `group` +already use, applied to every widget instead of a blessed subset. `DESCRIBE PAGE` +emits this form, replacing today's `pluggablewidget '' frame ( … )`. + +### Describing a widget is a statement + +```mdl +describe widget htmlelement; -- property table, enums, containers +list widgets; -- every widget with a definition +show references to widget htmlelement; -- which pages use it +``` + +`DESCRIBE WIDGET` is the statement that retires the drift risk: once the answer +is available in-language and against the live project, the generated `.md` stops +being the only source and can be regenerated from — or replaced by — it. + +### The ID form remains, as an escape hatch + +```mdl +widget 'com.acme.widget.Unlisted' w1 ( someProp: 'x' ) +``` + +For a widget whose definition is not loaded, or to be explicit. After slice 2 +this is rarely written and never emitted by `DESCRIBE`. + +### Design notes against `design-mdl-syntax.md` + +- **Reuse existing keywords first.** This proposal goes further: it stops + *adding* them. Today every new widget with an object list needs a new reserved + word — an unbounded cost paid in name collisions (#619's quoting escape hatch + exists for exactly this). +- **One way to do each thing.** Two spellings collapse to one: `combobox c (…)` + and `pluggablewidget 'com.mendix.widget.web.combobox.Combobox' c (…)` are the + same widget today, and `CUSTOMWIDGET` is a third spelling of the same + behaviour. +- **No implicit context**: a container name resolves against the parent widget's + own definition, which is as explicit as a qualified name. +- **One example is enough for an LLM** — and the example the generator already + emits becomes the correct one. + +### The one required doc change + +The generated example omits the **name**, which even a working slot requires: + +``` +tagcontentcontainer { ... } -- as generated today, rejected +tagcontentcontainer content { } -- correct +``` + +So the three child slots that *could* parse today are documented in a form that +cannot either. + +## Implementation Plan + +Six slices. Each ships alone; 1 is independent of the rest. + +### Slice 1 — Stop the bleeding + +Correct whether or not anything else lands. **Items 2–4 are implemented.** +Item 1 was dropped: the premise behind it turned out to be false (below). + +1. ~~**Ship the four missing built-in widgets.**~~ **Dropped — the premise was + wrong.** The draft assumed `events`, `fileuploader`, `googletag` and + `markdown` are bundled with Studio Pro and therefore unreachable to mxcli. + Measured instead: + + | | | + |---|---| + | A blank Mendix 11.13 project (`mx create-project`) | 33 widgets, **none of the four** | + | Installing File Uploader (Marketplace module 235351) | `widgets/` goes 33 → 34, `.mpk` present | + | `exec` of a page using it, straight after | **builds** — no `widget init` needed | + + So a widget whose package is absent is one **Studio Pro cannot use either**; + it is not a gap mxcli can paper over, and there is nothing to ship. The + moment the widget is usable at all, the `.mpk` is in the project and mxcli + picks it up on its own, because `initPluggableEngine` refreshes definitions + from installed packages before reading them. + + Two wrong turns are worth recording, since both looked settled at the time. + Their `.mpk`s are **not** inside `Mendix.Modeler.Core.dll` — that came from a + bare ID-string match, and the assembly has 690 embedded zips with *zero* + `widgets.mendix.com` hits. And embedding a `.def.json` would not have worked + anyway: `getOrGenerateTemplate` (`modelsdk/widgets/loader.go:215`) derives the + template from the `.mpk` **in `widgets/`**, so a def alone only moves the + error to `template not found: fileuploader` — verified before the premise + itself was checked, which is the lesson. The remaining item below is the + whole fix. + +2. **Fix that error message.** It currently says + `(run 'mxcli widget init -p app.mpr')` — a remedy that **provably cannot + work**, since `widget init` scans `widgets/` and these are not there. +3. **Generated examples include the name** on child slots. +4. **`()` is accepted.** `widgetPropertiesV3` requires at least one property, so + `container c ()`, `text t ()` and `pluggablewidget … pw ()` are all parse + errors while bare `pw` and `pw (x: 'y')` are fine. One-line grammar fix, found + in the same investigation. + +### Slice 2 — The widget keyword is def-driven — **implemented** + +5. **Done.** `widgetTypeV3` gains a generic `IDENTIFIER` alternative, ordered + last, so `htmlelement frame ( … )` works for every widget with a definition. +6. `DESCRIBE PAGE` emits the keyword form — **not done here**, see below. + +Not "grammar and visitor only", which was the estimate. The load-bearing half +was the validator, exactly as Open Question 1 warned: + +- The visitor records **which alternative matched** (`ast.WidgetV3.TypeIsGeneric`), + because `Type` alone cannot tell a typo from a built-in — both are a lowercase + string. It is taken from the parse tree, never by comparing the text against a + list of known names, which would reintroduce the list this proposal removes. +- **MDL-WIDGET25 grew a second branch** for a generic type that resolves to no + definition. Without it, measured: `htmlelemnt frame (tagName: 'div')` gave + *"0 errors, 1 warning"* — a warning about `tagName` — while the correct + spelling was completely clean. A typo had traded a parse error for a wrong + answer, which is worse than what it replaced. +- **MDL-WIDGET07 is suppressed** for a generic type that did not resolve. + Reporting its properties on top of the kind error points at the wrong token. + +Item 6 is left for its own change: `DESCRIBE PAGE` currently emits the +`pluggablewidget ''` form, which still round-trips, so this is a readability +improvement rather than a capability and does not belong in the same commit. + +### Slice 0 — The validator knows what a widget is (blocks slices 2–3) + +Report at `check` time what only `exec` catches today: an unknown widget kind or +id, and a container the parent's definition does not declare — each naming the +near misses. The detection already exists in `validateWidgetTreeIn`; only the +reporting is missing. Worth shipping on its own, and the thing that makes +slices 2–3 safe (Open Question 1). + +### Slice 3 — The widget body is def-driven — **implemented** + +7. **Grammar — simpler than proposed.** No parallel `pluggableBodyV3` / + `genericContainerV3` rule was needed: adding `keyword` beside `IDENTIFIER` in + the same last-ordered `widgetTypeV3` alternative covers containers, because a + container and a widget occupy the same position in a body. + + `keyword` is load-bearing, as the draft said: `attribute` lexes as the + `ATTRIBUTE` token and never as `IDENTIFIER`, so slice 2 alone cannot reach the + case that motivated the issue. + + **One ordering fix was required, and it is the finding of this slice.** + `pageBodyV3` listed `widgetV3` *first*. `SLOT`, `PLACEHOLDER` and `USE` are + all inside `keyword` (655 tokens), so the generic alternative swallowed them: + `slot body` became a widget of type `slot`, and `placeholder Main { … }` a + widget named `Main`. The specific alternatives now precede `widgetV3`. + +8. **Validator — done.** A container the parent does not declare is reported + against the parent, naming what it *does* declare: + + ``` + `attribut` is not a container of `htmlelement` — it declares: attribute, + event, tagcontentcontainer, tagcontentrepeatcontainer + ``` + + Inside a resolvable parent this beats "not a widget in this project", so the + generic branch routes to MDL-WIDGET26 when a parent definition is available + and to MDL-WIDGET25 when it is not. + +9. ~~**Delete the nine** from `widgetTypeV3`.~~ **Deferred, deliberately.** They + are no longer a capability boundary — anything else parses too — so what + remains is redundancy, not drift risk: a new container keyword needs no + maintenance because the generic path already accepts it. Removing them would + flip nine keywords to `TypeIsGeneric`, routing them through the generic + branch, where a container inside an *unresolvable* parent would newly report + MDL-WIDGET25. That is a false-positive class in exchange for tidiness. + +**Result.** The construct from the issue parses: + +```mdl +htmlelement frame (tagName: 'div') { + attribute a1 (attributeName: 'data-role', attributeValueType: 'expression') + tagcontentcontainer body { dynamictext t (Content: 'hi') } +} +``` + +Measured across the fixture's definitions: **50 of 50 containers authorable**, +from 16 of 46. With a project, properties are validated against the real +definition — an invented `attributeValueType: 'static'` is MDL-WIDGET08, *valid +values are expression, template*. + +### Slice 4 — `DESCRIBE WIDGET` / `LIST WIDGETS` + +10. An MDL statement returning what `mxcli widget describe` returns, from the + live project. This is what makes the generated `.md` optional rather than + load-bearing, and it is the slice that actually retires #1036's failure mode. + `LIST WIDGETS` follows the repo convention (`list`, not `show`, for new + commands); the existing `SHOW WIDGETS` is unrelated — it lists widget + *instances on pages*, not definitions. + +### Slice 5 — A `widget` edge in `CATALOG.REFS` — **implemented** + +11. ~~Emit one edge per widget instance on a page~~ — **one edge per page (or + snippet) x widget definition.** `show references to combobox` and + `show impact of htmlelement` work. Catalog work, not grammar; independent + of slices 2–4. + + Three details differ from the sketch above, each settled by measurement + rather than by the wording: + + - **Per container, not per instance.** DISTINCT collapses a page's seven + comboboxes into one edge, matching the four sibling widget projections in + `buildReferences`. Per-instance rows say nothing `SHOW REFERENCES` or + `SHOW IMPACT` can use, and `CATALOG.WIDGETS` already holds the instances. + + - **`TargetName` is the MDL name (`COMBOBOX`), not the widget ID.** The + dotted ID poisons the module-derived graph views, which take everything + before the first dot as the module: measured on `testdata/expr-checker`, + it invented a module `com` carrying 14 edges from three real modules, and + listed `com.mendix.widget.web.image.Image` in `graph_god_nodes` with + `ModuleName` `com`. The ID lives in `TargetId`. A widget is consequently + the **first non-dotted target** in the table (0 of 248 before), so + `graph_god_nodes` now excludes `WIDGET` from its asset side. + + - **Only widgets that resolve to a definition.** A built-in stores its BSON + `$Type` in the same column and has none, so it gets no edge — an edge + should point at something describable. + + The syntax in the original sketch (`show references to widget htmlelement`) + was not needed: `show references to ` already parses both a bare word + and a dotted ID, so no grammar changed. It did need one fix on the executor + side — the stored name is SHOUTED while MDL keywords are written in lower + case, so the natural spelling answered "(no references found)", a wrong + answer rather than a missing one. + + Not done, and deliberately: a widget definition is **not** added to + `objects`, so an unused `.mpk` does not appear in `GRAPH_DEAD_ASSETS`. The + anti-join that answers it is documented in `catalog-schema.md` instead. + +### Slice 6 — `PLUGGABLEWIDGET` → `WIDGET` (and collapse `CUSTOMWIDGET`) + +12. Deliberately last: after slice 2 the ID form is rarely written and never + emitted, so this is a readability change on an escape hatch rather than a + headline. Its real value is collapsing `CUSTOMWIDGET` — which already takes + the identical code path and writes the identical BSON — into one keyword, + removing a genuine "two ways to do one thing". + + Old spellings stay accepted (never emitted), so 164 occurrences across 42 + example, skill and doc files keep working and can be migrated at leisure. + + **The cost to weigh**: `WIDGET` already appears in + `ALTER/DESCRIBE STYLING ON PAGE … WIDGET name`, where it means *the widget + named X* rather than *a widget of type X*. The positions are disjoint so it + parses, and both readings are still "widget" — the same way `PAGE` is both + declared and referenced — but it is the argument against, and it should be + made explicitly rather than discovered. + +### Files to modify/create + +| File | Change | +|------|--------| +| `modelsdk/widgets/definitions/{events,fileuploader,googletag,markdown}.def.json` | **new** — the four missing built-ins (slice 1) | +| `mdl/executor/widget_engine.go` | correct the `no definition for widget` remedy (slice 1) | +| `mdl/executor/widget_defs.go` | emit the name in generated examples (slice 1) | +| `mdl/grammar/domains/MDLPage.g4` | `()` accepted (1); identifier widget type (2); `pluggableBodyV3` + `genericContainerV3`, remove the nine (3) | +| `mdl/visitor/visitor_page_v3.go` | set `Type` from an identifier widget type and a generic container | +| `mdl/executor/validate_widgets.go` | resolve containers against the def; suggest near-misses | +| `mdl/executor/cmd_pages_describe_output.go` | emit the keyword form (slice 2) | +| `mdl/grammar/domains/MDLCatalog.g4`, `mdl/ast/`, `mdl/executor/cmd_widgets.go` | `DESCRIBE WIDGET` / `LIST WIDGETS` (slice 4) | +| `mdl/catalog/builder_references.go` | `widget` edge (slice 5) | +| `mdl-examples/doctype-tests/`, `mdl-examples/bug-tests/1036-*.mdl` | the reporter's four cases; HTML Element attributes + events + a child slot | +| `docs/01-project/MDL_QUICK_REFERENCE.md`, `cmd/mxcli/syntax/features_*.go` | the widget vocabulary is per-project, not fixed | + +## Version Compatibility + +None. MDL-side only: no new BSON, no Mendix API, no feature-registry entry, no +`checkFeature()` gate. A widget's object lists come from its own `.mpk`, so +version differences are already carried by the def +(`PROPOSAL_multi_version_pluggable_widgets.md`). + +## Test Plan + +- **The guard that makes this unreintroducible.** Derive every widget keyword, + object-list keyword and child-slot keyword from every `.def.json` in the + fixture, generate a minimal page per keyword, and assert it parses. This test's + absence *is* the bug: two lists and nothing comparing them. Present numbers — + 16 of 46 containers, and 1 of 33 widget keywords — become all of them. +- **The reporter's four cases**, verbatim, in `mdl-examples/bug-tests/`. +- **Round-trip**: `create` → `describe` → `exec` for a widget with both an object + list and a child slot, asserting the description re-parses **and** that + `DESCRIBE` now emits the keyword form. +- **`mx check` at 0 errors** on a project carrying an HTML Element with + `attributes`, plus a Studio Pro open. The BSON path is unchanged, but the claim + that it is unchanged deserves one measurement. +- **Error-quality tests** (the regression risk): an unknown container names the + valid ones for *that* widget; an unknown widget keyword names near-misses; a + typo'd built-in widget still fails informatively. +- **Controls, per CLAUDE.md.** Reverting the validator must make the typo case + report the raw parse error; reverting the grammar must reproduce + `mismatched input 'attribute' expecting '}'`. A test that only passes against + fixed code has not been shown to detect anything. + +## Open Questions + +1. ~~**Does error quality actually survive?**~~ **Settled: no, not as things + stand — and the fix is a prerequisite slice, not a risk to accept.** + + The question assumed the validator would need to match what the parser + catches. Measured, the validator catches **less than assumed**, and the gap + is already open for every case that reaches it today: + + | written | today's verdict | + |---|---| + | `contaner c1 (…)` — typo'd widget kind | **parse error** (the parser is the allow-list) | + | `pluggablewidget 'com.acme.NotAWidget' w1` — unknown widget | `check` **passes**; fails at `exec` | + | `group g1 (…)` inside HTML Element — real keyword, wrong widget | `check` **passes**; fails at `exec` | + + So **`widgetTypeV3` is currently the widget-kind validator.** The validator + has no independent notion of "is this a real widget kind" and does not need + one, because nothing else can parse. Slices 2–3 remove that enforcement, so + as written they would move *every* container mistake into the hole the last + two rows already occupy: `check` green, failure at `exec`. + + **But the detection is already computed.** `validateWidgetTreeIn` holds the + parent's declared object lists and looks the child up in them + (`mapping := parentObjectLists[strings.ToUpper(w.Type)]`), and + `lookupWidgetDef` says whether the type is a known widget. Nothing *reports* + when both miss — the branch routes to `validateStaticWidgetUnknownProps`, + which checks properties of a presumed static widget rather than questioning + the kind. + + That reframes the work. Closing the hole is an improvement **today**, + independent of any grammar change: it makes `check` catch two mistakes that + currently reach `exec`. And once `check` reports them, this question is + answered affirmatively **by construction**, because the semantic error exists + before the parse error is given up. + + **Resolution: a new Slice 0 — "the validator knows what a widget is" — lands + before slices 2–3, and they are blocked on it rather than on a decision.** + It must report, with near-miss suggestions: an unknown widget kind or id, and + a container the parent's definition does not declare. Its own control is that + a *correct* widget and a *correct* container stay silent. + +2. ~~**How far does the ambiguity reach?**~~ **Settled by measurement.** Built + the grammar and diffed `mxcli check` output across all 515 scripts in + `mdl-examples/`, at each step: + + | | verdict changes | message changes | + |---|---|---| + | `IDENTIFIER` (slice 2) | 0 of 515 | 0 | + | `+ keyword` (slice 3, 655 tokens) | 0 of 515 | 0 | + + Ordering the alternative last contains it, and ALL(\*) resolves the rest. + + **But the diff was not sufficient, and that is the more useful finding.** It + compares DIAGNOSTICS, and the real damage was to the AST: `slot body` and + `placeholder Main { … }` were silently reparsed as widgets, still exiting 0. + Two visitor unit tests caught what 515 scripts could not. A corpus diff of + `check` output cannot see a construct that parses into the wrong shape. + + Running it also required fixing the tool: three validators emitted one + violation per property while ranging over a map, so two runs of the *same* + binary disagreed on 11 of 515 scripts — a noise floor larger than the signal. +3. **Should child slots stay named?** Consistency says yes and existing documents + require it. But a slot is a fixed property, not a repeating item, so its name + is never referenced — `tagcontentcontainer { }` reads better and is what the + generator emits today. Allowing both would be two spellings. + Recommend: keep the name, fix the doc. +4. **Does slice 4 make the generated `.md` redundant?** It should, for an agent + that can run `mxcli`. It does not for one reading a repo cold, which is the + case `widget init` was built for. Likely answer: keep generating, but from the + same code path `DESCRIBE WIDGET` uses, so they cannot disagree. +5. ~~**What is a widget edge's source granularity?**~~ **Settled by slice 5: per + (container, widget definition).** Per instance says nothing `SHOW REFERENCES` + or `SHOW IMPACT` can use — both list sources, and `CATALOG.WIDGETS` already + holds the instances — so the extra rows buy nothing at any project size. It + also matches the four sibling widget projections in `buildReferences`, which + have collapsed with DISTINCT all along. + + The question that turned out to matter was not granularity but **what to put + in `TargetName`**, which the draft did not ask. The dotted widget ID poisons + every module-derived graph view; the MDL name does not. See slice 5. +6. ~~**The built-in census.**~~ **Moot — the premise it rested on was wrong.** + It asked whether the four widget IDs scraped out of `Mendix.Modeler.Core.dll` + were a complete list of Studio Pro's bundled widgets. Slice 1 established + they are not bundled at all: a blank 11.13 project ships 33 widgets, none of + them, and a widget whose `.mpk` is absent is one Studio Pro cannot use + either. There is no list to complete. diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 5ad06d2367..ce98eec667 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -135,6 +135,7 @@ for display in this README): | [Replace Generated Playwright Tests with playwright-cli](proposal-playwright-cli.md) | Draft | The current approach (documented in proposal-playwright-testing.md) has Claude Code generate TypeScript test files (.spec.ts), then run them | | [Self-Describing Syntax Feature Registry](syntax-feature-registry.md) | Draft | Branch: research/recursive-help-discovery | | [Structured description of irreducible microflow graphs](PROPOSAL_structured_microflow_description.md) | Draft | DESCRIBE MICROFLOW renders a microflow's control flow as nested if/then/else. | +| [Widgets as first-class MDL, not a second dialect](PROPOSAL_def_driven_widget_bodies.md) | Draft | A widget is named, described and tracked differently from every other MDL extension point, and resolves against hardcoded grammar lists rather than the project. | | [Translations — preserve, describe, author, and auto-translate](PROPOSAL_translations.md) | Partial | A Mendix app ships its user-visible strings in every language it supports. | | [Version-Aware Agent Support](PROPOSAL_version_aware_agent_support.md) | Draft | Three use cases require mxcli to be version-aware at the MDL level: | | [warm dev loop — Docker-free run and iPad split-screen preview](PROPOSAL_mxcli_dev_warm_loop.md) | Draft | Relates to: PROPOSAL_check_mxbuild_gap_heuristics.md (the static-check gate that | diff --git a/mdl-examples/bug-tests/1036-def-driven-widget-body.mdl b/mdl-examples/bug-tests/1036-def-driven-widget-body.mdl new file mode 100644 index 0000000000..111383ed4f --- /dev/null +++ b/mdl-examples/bug-tests/1036-def-driven-widget-body.mdl @@ -0,0 +1,80 @@ +-- mendixlabs/mxcli#1036 — slices 2 and 3 of PROPOSAL_def_driven_widget_bodies.md +-- +-- `mxcli widget init` generates widget documentation for agents to follow. Fed +-- back into `mxcli check` verbatim, the generated htmlelement.md failed on the +-- first line of its own example: +-- +-- line 4:4 mismatched input 'tagcontentcontainer' expecting '}' +-- +-- The doc generator derived a keyword for every object list and child slot in a +-- widget definition; the grammar accepted nine hardcoded ones. Two lists, and +-- nothing comparing them. Measured on the fixture's definitions: 16 of 46 +-- documented constructs parsed. It is 50 of 50 now. +-- +-- The cost was not the wasted sessions. Unable to express the `attributes` list +-- below, the reporter's team fell back to tagContentHTML, which executes +-- same-origin — the exact risk the sandbox they wanted was for. +-- +-- Nothing here is a new keyword. Every name in this file comes from a widget's +-- own definition, which is why `mxcli widget describe htmlelement` lists them. + +create module DefDriven; + +-- A widget named by its own MDL name. `htmlelement` was never in the grammar's +-- widget-type list, though the builder has always resolved it: the page builder +-- tries widgetRegistry.Get(ToUpper(type)) FIRST. Only ANTLR needed a token. +create page DefDriven.HtmlElementBody ( + Title: 'HTML Element with attributes', + Layout: Atlas_Core.Atlas_Default +) { + htmlelement frame (tagName: 'div') { + + -- Object list. `attribute` lexes as the ATTRIBUTE keyword token and never + -- as IDENTIFIER, so accepting a bare identifier as a widget type is NOT + -- enough to reach it — this is the case slice 3 exists for. + attribute a1 ( + attributeName: 'data-role', + attributeValueType: 'expression' + ) + attribute a2 ( + attributeName: 'aria-label', + attributeValueType: 'expression' + ) + + -- A second object list on the same widget. + event e1 ( + eventName: 'onClick' + ) + + -- Child slot: a fixed property holding widgets, not a repeating item. + tagcontentcontainer body { + dynamictext greeting (Content: 'Hello from inside an HTML Element') + } + } +} + +-- A pluggable widget with no properties at all. `()` on a def-driven widget +-- parses for the same reason it does on a built-in (slice 1). +create page DefDriven.BareWidget ( + Title: 'Bare', + Layout: Atlas_Core.Atlas_Default +) { + htmlelement plain () +} + +-- Containers on widgets that already had keywords keep working unchanged — +-- the enumerated alternatives are still tried first, so nothing about an +-- ordinary widget body moved. +create page DefDriven.StillWorks ( + Title: 'Unchanged', + Layout: Atlas_Core.Atlas_Default +) { + accordion acc () { + group g1 (headerText: 'Section one') { + dynamictext t1 (Content: 'inside a group') + } + } + container plain (Class: 'row') { + dynamictext t2 (Content: 'inside a container') + } +} diff --git a/mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl b/mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl new file mode 100644 index 0000000000..3b1d9eff8e --- /dev/null +++ b/mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl @@ -0,0 +1,54 @@ +-- Slice 0 of PROPOSAL_def_driven_widget_bodies.md: the validator knows what a +-- widget is. Both statements below used to pass `mxcli check` and fail only at +-- `exec`. +-- +-- Until now the GRAMMAR was the widget-kind validator: `widgetTypeV3` is an +-- allow-list, so an unknown kind could not parse and the validator never needed +-- an independent notion of one. Two mistakes are not keywords, so neither was +-- caught: +-- +-- MDL-WIDGET25 an explicit widget id that resolves to nothing +-- MDL-WIDGET26 a real container keyword on a parent that has no such container +-- +-- Reporting them is worth doing on its own. It is also what makes slices 2-3 +-- safe: those give up the parser's enforcement, so the semantic check has to +-- exist first (Open Question 1). +-- +-- Both rules are deliberately silent when they cannot be sure: +-- * an id whose .mpk IS installed is real, just not extracted yet — the +-- registry used by `check` reads .mxcli/widgets/ and does NOT refresh from +-- installed packages, so without this guard every widget in a project that +-- never ran `widget init` would be called unknown +-- * a container is never judged against a parent whose definition could not +-- be resolved, for the same reason +-- +-- NOT a .fail.mdl, deliberately. `make check-mdl` runs `check` WITHOUT a +-- project, and both rules need one: +-- +-- MDL-WIDGET25 cannot tell an unknown widget from one installed in a project +-- it cannot see, so with no -p it stays silent +-- MDL-WIDGET26 needs the PARENT's definition resolved, and without a project +-- the registry holds only the nine embedded widgets +-- +-- So this file passes check with no project and reports both rules with one. +-- That is the case the Makefile warns about (#891, #892): naming it .fail.mdl +-- would report "negative test unexpectedly passed" and make a working rule look +-- regressed. The rules are covered by unit tests in validate_widget_kind_test.go +-- instead; this file is the human-readable repro. +-- +-- Run it against a project to see them: +-- mxcli check mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl -p app.mpr + +create page Bug1036.SliceZero ( + title: 'Slice 0', + layout: Atlas_Core.Atlas_Default, + folder: 'Bug1036' +) { + -- MDL-WIDGET25: no definition, and no package in widgets/ either. + pluggablewidget 'com.acme.widget.NotAWidget' w1 (someProp: 'x') + + -- MDL-WIDGET26: `group` is Accordion's container, not HTML Element's. + pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' h (tagName: 'div') { + group g1 (headerText: 'HTML Element has no groups') + } +} diff --git a/mdl-examples/bug-tests/1036-widget-discovery-honesty.mdl b/mdl-examples/bug-tests/1036-widget-discovery-honesty.mdl new file mode 100644 index 0000000000..a2b2d2813b --- /dev/null +++ b/mdl-examples/bug-tests/1036-widget-discovery-honesty.mdl @@ -0,0 +1,40 @@ +-- Slice 1 of PROPOSAL_def_driven_widget_bodies.md — the parts of +-- mendixlabs/mxcli#1036 that need no design work. +-- +-- Three independent defects, all found while reproducing the report: +-- +-- 1. `()` was a parse error on EVERY widget kind. `container c` parsed and +-- `container c (x: 'y')` parsed, but `container c ()` — what an LLM writes +-- for a widget that needs no properties — failed at the `)` with an error +-- that read as though the widget itself were wrong. +-- +-- 2. The `no definition for widget` error always said "run 'mxcli widget init +-- -p app.mpr'". For a widget Studio Pro BUNDLES rather than installs (File +-- Uploader, Events, Google Tag, Markdown viewer) that is worse than +-- unhelpful: widget init scans widgets/, the .mpk is not there, and +-- re-running it can never help. It is now branched on whether the package is +-- actually installed — the same question FindMPK answers for the template +-- loader. +-- +-- 3. `mxcli widget init` generated child-slot examples with NO NAME +-- (`tagcontentcontainer { … }`), which even a working slot rejects — so the +-- three slots that DID parse were documented in a form that could not. +-- Names are now emitted and numbered, since two `slot1`s on one page would +-- collide. +-- +-- Not in this file: the capability gap itself (30 of 46 documented constructs +-- do not parse). That is slices 2-3 of the proposal. + +create page Bug1036.EmptyProps ( + title: 'Empty property lists', + layout: Atlas_Core.Atlas_Default, + folder: 'Bug1036' +) { + -- All three spellings are now accepted, and mean the same thing. + container outer () { + container inner { + dynamictext note ( Content: 'both spellings parse' ) + } + dynamictext empty () + } +} diff --git a/mdl-examples/bug-tests/assoc-datasource-page-level.mdl b/mdl-examples/bug-tests/assoc-datasource-page-level.mdl new file mode 100644 index 0000000000..60ec8c3719 --- /dev/null +++ b/mdl-examples/bug-tests/assoc-datasource-page-level.mdl @@ -0,0 +1,102 @@ +-- An association datasource at PAGE level typed its rows as the entity it was +-- navigating AWAY from (mendixlabs/mxcli#1045): +-- +-- datagrid gA (datasource: $Customer/Bench.Order_Customer) { … OrderNo … } +-- mx check -> [CE1613] "The selected attribute 'Bench.Customer.OrderNo' +-- no longer exists." at Columns (1/1) of data grid 2 'gA' +-- +-- The destination entity is resolved as "the end opposite the context", and the +-- context used was the ENCLOSING data container's entity. That is the right +-- answer inside a data view and empty at page level, where nothing encloses the +-- widget — so neither end matched and the last-resort fallback returned the +-- association's TO side, which is the entity the grid started from. Every column +-- was then bound against it. +-- +-- The report's own diagnosis was right and is worth keeping: the SAME path +-- inside a data view was already correct, which is what localised the bug to +-- page level. A named context variable answers the question directly — +-- `$Customer/…` traverses from whatever $Customer holds, enclosed or not. +-- +-- Verify: +-- +-- mxcli exec assoc-datasource-page-level.mdl -p app.mpr +-- mx check -p app.mpr -- 0 errors; before the fix, one CE1613 on P_PageLevel +-- +-- Unlike the member-resolution examples in this directory, this one does NOT +-- need exec-before-check: the defect is in what the WRITER stores, so the proof +-- is `mx check` on the result rather than a report from `mxcli check`. + +create module AssocDS; +/ + +create or modify entity AssocDS.Customer ( + Name: String(200) +); +/ + +create or modify entity AssocDS.Order ( + OrderNo: String(50) +); +/ + +-- Order is the FROM end and Customer the TO end, so traversing FROM a Customer +-- reaches Orders. That direction is the one that was wrong. +create or modify association AssocDS.Order_Customer + from AssocDS.Order to AssocDS.Customer type reference; +/ + +-- PAGE LEVEL: nothing encloses the grid, so the only thing that can say what the +-- association is traversed from is `$Customer`. This is the reported case. +create or replace page AssocDS.P_PageLevel +( + params: { $Customer: AssocDS.Customer }, + title: 'Orders for a customer', + layout: Atlas_Core.Atlas_Default +) +{ + datagrid gridOrders (datasource: $Customer/AssocDS.Order_Customer) { + column cNo (attribute: OrderNo, caption: 'No') + } +} +/ + +-- CONTROL: the same traversal INSIDE a data view. This was already correct, and +-- must stay correct — the fix only adds a better answer where a variable is +-- named, and leaves the enclosing-entity path alone. +create or replace page AssocDS.P_InDataView +( + params: { $Customer: AssocDS.Customer }, + title: 'Same thing, enclosed', + layout: Atlas_Core.Atlas_Default +) +{ + dataview dvCustomer (datasource: $Customer) { + datagrid gridOrders2 (datasource: $currentObject/AssocDS.Order_Customer) { + column cNo2 (attribute: OrderNo, caption: 'No') + } + } +} +/ + +-- CONTROL: the REVERSE direction. Traversing from an Order reaches its Customer, +-- so the widget here binds Name. Without this a fix that always returned the +-- FROM end would look correct on the two pages above. +-- +-- Note the widget: a DATA VIEW, not a grid. Traversing a Reference from its FROM +-- end yields ONE object, and Mendix rejects a grid over it with a separate and +-- correct complaint — [CE8812] "A grid association path must result in a list." +-- That is about cardinality rather than about which entity was resolved, and +-- writing it as a grid here would have made this control fail for the wrong +-- reason. +create or replace page AssocDS.P_Reverse +( + params: { $Order: AssocDS.Order }, + title: 'The other way', + layout: Atlas_Core.Atlas_Default +) +{ + dataview dvCustomer (datasource: $Order/AssocDS.Order_Customer) { + textbox tName (label: 'Customer', attribute: Name) + } +} +/ diff --git a/mdl-examples/bug-tests/create-shape-checked-without-a-project.mdl b/mdl-examples/bug-tests/create-shape-checked-without-a-project.mdl new file mode 100644 index 0000000000..d46341a341 --- /dev/null +++ b/mdl-examples/bug-tests/create-shape-checked-without-a-project.mdl @@ -0,0 +1,123 @@ +-- Three things `mxcli check` accepted that exec or the build refused. All three +-- are answerable from the STATEMENT alone, so they are reported with no project +-- — a divergence that only `-p` catches is still a divergence. +-- +-- 1. An UNQUALIFIED CREATE (mendixlabs/mxcli#1050): +-- +-- create association Order_Probe from Bench.Order to Bench.Customer … +-- mxcli check --references -> "All references valid", exit 0 +-- mxcli exec -> Error: module name is required: objects must +-- be created within a module +-- +-- The cost is not the error, it is that exec is NOT TRANSACTIONAL: the +-- statements before this one are already applied when it fails, and +-- re-running the script then hits "already exists" on them. Now MDL074. +-- +-- 2. `RETURNS void AS $x` (mendixlabs/mxcli#1041): +-- +-- CREATE MICROFLOW … RETURNS void AS $result BEGIN COMMIT $C; END; +-- mx check -> [CE0109] "Undefined variable 'result'." at End event +-- +-- An alias names the variable a flow returns, so pairing it with void is a +-- contradiction — and mxcli believed the alias, writing `return $result` into +-- a flow that has no such variable. Refused rather than repaired: emitting a +-- bare `return` would also build, but the author who wrote an alias meant to +-- return something, and silently dropping it makes the flow return nothing +-- while its source still says otherwise. Now MDL075. +-- +-- 3. A CODE-LESS, LOCATION-LESS parse error (mendixlabs/mxcli#1042): +-- +-- IF empty($Orders) THEN … +-- ✗ Unexpected token after expression … [] <- no code, no document +-- +-- `empty` is a Mendix KEYWORD, not a function, so the parser consumed it and +-- stopped at the '('. The hint now carries a code (E014) and the microflow it +-- is in, and its fix line names that cause first instead of sending people +-- hunting for a glued 'emptyor' that was not there. The line and column stay +-- offsets into the expression fragment rather than into the file, so the flow +-- name is the locator — stated because it is a real limit, not an oversight. +-- +-- WHICH TIER EACH ONE RUNS IN — measured, because they are not the same and the +-- difference decides how you reproduce them: +-- +-- MDL074, MDL075 no project. `mxcli check file.mdl` reports them. +-- E014 needs -p, and needs the file to be otherwise clean. +-- Expression checking is the catalog-backed tier and runs +-- AFTER the reference check, which EXITS on its first error — +-- so one unrelated mistake anywhere in the file hides every +-- expression hint in it. Measured while writing this example: +-- a control microflow here had `set $out = …` without +-- declaring $out, and that one reference error made the E014 +-- perturbation below look like it did not fire at all. +-- +-- Verify: +-- +-- mxcli check create-shape-checked-without-a-project.mdl +-- mxcli exec create-shape-checked-without-a-project.mdl -p app.mpr +-- mx check -p app.mpr -- 0 errors +-- +-- Then perturb, one at a time: +-- +-- create entity CreateShape.Thing … -> create entity Thing … +-- (no -p needed) MDL074: entity "Thing" has no module … +-- +-- returns void -> returns void as $result +-- (no -p needed) MDL075: … cannot be paired with void … CE0109 +-- +-- IF $Things = empty THEN -> IF empty($Things) THEN +-- (needs -p, and needs the rest of the file to be reference-clean — see the +-- tier note above) E014: unexpected token after the expression … +-- at CreateShape.ACT_Count + +create module CreateShape; +/ + +create or modify entity CreateShape.Thing ( + Code: String(20) +); +/ + +create or modify entity CreateShape.Other ( + Label: String(20) +); +/ + +-- CONTROL for MDL074: a qualified association is exactly what the rule wants, +-- and a `create module` — which has nothing to be qualified BY — must never be +-- reported. Both appear in this file. +create or modify association CreateShape.Thing_Other + from CreateShape.Thing to CreateShape.Other type reference; +/ + +-- CONTROL for MDL075: `RETURNS void` on its own is the spelling the report +-- itself identified as round-tripping cleanly. +create or modify microflow CreateShape.ACT_Void ( $Thing: CreateShape.Thing ) +returns void +begin + COMMIT $Thing; +end; +/ + +-- CONTROL for MDL075: a typed return WITH an alias is the ordinary shape. A rule +-- that flagged the alias itself would break far more than it fixed. +create or modify microflow CreateShape.ACT_Typed () +returns string as $out +begin + DECLARE $out String = 'done'; + RETURN $out; +end; +/ + +-- CONTROL for E014: the two correct ways to ask whether a list is empty. +-- `empty` is a keyword, so it goes on the right of a comparison. +create or modify microflow CreateShape.ACT_Count () +begin + RETRIEVE $Things FROM CreateShape.Thing; + IF $Things = empty THEN + LOG INFO 'none'; + END IF; + IF length($Things) = 0 THEN + LOG INFO 'still none'; + END IF; +end; +/ diff --git a/mdl-examples/bug-tests/enum-add-value-if-not-exists.mdl b/mdl-examples/bug-tests/enum-add-value-if-not-exists.mdl new file mode 100644 index 0000000000..f1eb11b677 --- /dev/null +++ b/mdl-examples/bug-tests/enum-add-value-if-not-exists.mdl @@ -0,0 +1,66 @@ +-- `alter enumeration … add value` had no IF NOT EXISTS, so a script that adds +-- an enumeration value was not re-runnable (ako/mxcli-rest FINDINGS #60). +-- +-- The harm is not the error. It is that `exec` STOPS at the failing statement, +-- so one already-present value leaves every LATER statement in the file +-- unapplied — a script that half-ran, reported an error about an enumeration, +-- and silently skipped the rest. Measured with the bare form first and a +-- guarded add second: the second value was absent afterwards. +-- +-- A defensive drop-then-add is not a workaround, which is why the guard has to +-- exist: the drop fails when the value is absent and the add when it is +-- present, so neither ordering is re-runnable on its own. +-- +-- ONE THING TO KNOW BEFORE READING THE SCRIPT. `create or modify enumeration` +-- REPLACES the value list rather than merging into it (measured: an enum with +-- three values, re-declared with one, keeps only that one). So an ALTER that +-- adds a value is undone by any later re-declaration of the enum, and a script +-- where the CREATE precedes the ALTER re-adds rather than skipping — which is +-- exactly why the values below are declared once and the ALTERs target values +-- the CREATE already lists. +-- +-- Verify — the point is that running this file TWICE behaves identically, so +-- run it twice: +-- +-- mxcli exec enum-add-value-if-not-exists.mdl -p app.mpr +-- mxcli exec enum-add-value-if-not-exists.mdl -p app.mpr +-- mx check -p app.mpr -- 0 errors +-- +-- Both runs print, for the guarded add: +-- Value 'Outbound' already exists on enumeration EnumGuard.Lane — skipped +-- and carry on to the statements below it. +-- +-- CONTROL: drop the `if not exists` from that ALTER and run again. It fails with +-- value 'Outbound' already exists on enumeration EnumGuard.Lane — use +-- 'add value if not exists' to make the script re-runnable +-- and the ALTERs after it do not run. + +create module EnumGuard; +/ + +create or modify enumeration EnumGuard.Lane ( + Inbound 'Inbound', + Outbound 'Outbound' +); +/ + +-- The guard, on a value that is already there. Without `if not exists` this +-- statement ends the run. +alter enumeration EnumGuard.Lane add value if not exists Outbound caption 'Outbound'; +/ + +-- The statement that proves the run was not truncated: it only lands if the one +-- above did not stop the script. Re-declaring the enum above would drop it +-- again, so this value exists only between here and the next CREATE. +alter enumeration EnumGuard.Lane add value if not exists Internal caption 'Internal'; +/ + +-- The DROP twin. A re-run finds the value already gone and skips rather than +-- reporting "enumeration value not found" and halting. +alter enumeration EnumGuard.Lane drop value if exists Retired; +/ + +-- CONTROL: an unguarded ALTER that is naturally idempotent still works +-- unchanged — MODIFY re-captions the same value however many times it runs. +alter enumeration EnumGuard.Lane modify value Inbound caption 'Inbound / Received'; +/ diff --git a/mdl-examples/bug-tests/expression-kinds-checked-at-check.mdl b/mdl-examples/bug-tests/expression-kinds-checked-at-check.mdl new file mode 100644 index 0000000000..fc8e5b9bbf --- /dev/null +++ b/mdl-examples/bug-tests/expression-kinds-checked-at-check.mdl @@ -0,0 +1,103 @@ +-- Two expression defects that passed `check --references`, passed `exec`, and +-- arrived as CE0117 "Error(s) in expression" at the far end of a build. +-- +-- 1. A BARE WORD as a member's value (mendixlabs/mxcli#1044): +-- +-- CHANGE $Order (Status = Closed); -- Closed is not an expression +-- mx check -> [CE0117] at Change object activity 'Change 'Order' (Status)' +-- +-- Mendix expressions have no bare identifiers — a value is a literal, a +-- $variable, a qualified name or a function call — so the checker read +-- `Closed` as a variable reference, it resolved to nothing, and the kind came +-- out Unknown. Unknown is tolerated everywhere by design, which is exactly +-- how this slipped through. Now E013. +-- +-- 2. A NON-STRING log template parameter (mendixlabs/mxcli#1043): +-- +-- LOG WARNING 'qty {1}' WITH ({1} = $Order/Qty); +-- mx check -> [CE0117] at Log message activity 'Log message (warning)' +-- +-- The root cause was one line wide: exprcheck's slot-expectation table had +-- existed with NOTHING READING IT — slotKind() was defined and never called +-- — so `LogStmt.Message: {Kind: KindString}` constrained nothing either, and +-- `LOG WARNING 42` passed too. Now E009. +-- +-- WHICH KINDS ARE ACTUALLY REJECTED, measured on 11.13.0 by executing four +-- microflows and building them — three errors for four parameters: +-- +-- {1} = $O/OrderNo (String) clean +-- {1} = $O/Qty (Integer) CE0117 +-- {1} = $O/Price (Decimal) CE0117 +-- {1} = $O/Active (Boolean) CE0117 +-- {1} = $O/Due (DateTime) CE0117 +-- {1} = $Customer (Object) CE0117 +-- {1} = toString(…) clean +-- +-- The report generalised this to "the writer's template/parameter emission is +-- wrong, not the caller's type". It is not: toString(...) around the same value +-- builds cleanly, so the writer is fine and Mendix simply wants a String. +-- +-- Verify — everything below must PASS: +-- +-- mxcli exec expression-kinds-checked-at-check.mdl -p app.mpr +-- mxcli check expression-kinds-checked-at-check.mdl -p app.mpr --references +-- mx check -p app.mpr -- 0 errors +-- +-- Then perturb, one at a time: +-- +-- Status = 'Closed' -> Status = Closed +-- E013: a bare word is not a Mendix expression … Fix: quote it ('Closed'), +-- write $Closed, or qualify it (Module.Enum.Value) +-- +-- {1} = toString($O/Qty) -> {1} = $O/Qty +-- E009: this position requires String, but the expression has kind Integer + +create module ExprKinds; +/ + +create or modify entity ExprKinds.Customer ( + Name: String(200) +); +/ + +create or modify entity ExprKinds.Order ( + OrderNo: String(50), + Status: String(50), + Qty: Integer +); +/ + +-- Every value here IS a Mendix expression, which is the point: these are the +-- shapes that must not be reported. +create or modify microflow ExprKinds.ACT_Close ( $Order: ExprKinds.Order ) +begin + CHANGE $Order ( + "Status" = 'Closed', -- a quoted literal + "OrderNo" = toString($Order/Qty) -- a call + ); + COMMIT $Order; +end; +/ + +create or modify microflow ExprKinds.ACT_Log ( $Order: ExprKinds.Order, $Customer: ExprKinds.Customer ) +begin + -- A String attribute needs no conversion. + LOG INFO 'order {1}' WITH ({1} = $Order/OrderNo); + -- Anything else does. This is the line the report was about. + LOG INFO 'qty {1}' WITH ({1} = toString($Order/Qty)); + -- A string literal parameter, and the message itself. + LOG WARNING 'for {1}' WITH ({1} = $Customer/Name); +end; +/ + +-- CONTROL for the scope of E013. A bare name NESTED in a list-operation +-- predicate is LEGAL — `Status` there resolves against the item under test — so +-- a rule that fired on any bare identifier would reject this working microflow. +-- Only a bare word that is the WHOLE value of a member can only be a mistake. +create or modify microflow ExprKinds.ACT_Filter () +begin + RETRIEVE $All FROM ExprKinds.Order; + $Open = FILTER($All, Status = 'Open'); + LOG INFO 'filtered'; +end; +/ diff --git a/mdl-examples/bug-tests/image-widget-hidden-maxheight.mdl b/mdl-examples/bug-tests/image-widget-hidden-maxheight.mdl new file mode 100644 index 0000000000..4d39e35c1b --- /dev/null +++ b/mdl-examples/bug-tests/image-widget-hidden-maxheight.mdl @@ -0,0 +1,88 @@ +-- Every mxcli-authored image widget failed the build (mxcli-ledger §142). +-- +-- On a project at 0 errors, one image widget on one new page: +-- +-- [error] [CE0463] "The definition of this widget has changed. Update this +-- widget by right-clicking it and selecting 'Update widget'…" +-- at Image 'imgProbe' +-- +-- A field-level diff of Atlas' own brand image against a describe → rename → +-- exec copy of it, GUIDs masked, differs in ONE line of 1480: +-- +-- === ONLY IN ATLAS === Object/Properties/21/Value/PrimitiveValue = '250' +-- === ONLY IN MINE === Object/Properties/21/Value/PrimitiveValue = '0' +-- +-- Through its TypePointer that property is `maxHeight`. MDL has no keyword for +-- it, so it is never written and keeps whatever the widget TEMPLATE captured — +-- 0, against Image 1.6.0's declared 250. +-- +-- WHY THE FIRST FIX FOR THIS DID NOTHING, which is the part worth keeping. The +-- writer already default-values a hidden property MDL cannot name, and +-- `maxHeight` IS hidden here ("hidden when maxHeightUnit = none"). The +-- condition was evaluated against `maxHeightUnit`'s DECLARED default, which +-- Image 1.6.0 states as "pixels" — so the rule read `maxHeight` as visible and +-- wrote nothing. But `maxHeightUnit` is unmapped too, so the document actually +-- gets the template's "none". The question "will this property be hidden?" has +-- to be asked of the configuration that will be WRITTEN, not of the one the +-- package declares. Its test had encoded the default as "none", so it passed +-- against an input that does not exist. +-- +-- Ground truth, measured across the 69 Image widgets of a real 11.14.0 project: +-- all 65 that carry a `maxHeight` store 250, at every combination of heightUnit +-- and maxHeightUnit. The one mxcli wrote was the only outlier. +-- +-- Verify: +-- +-- mxcli exec image-widget-hidden-maxheight.mdl -p app.mpr +-- mx check -p app.mpr -- 0 errors +-- +-- Use `mx check` DIRECTLY, not `mxcli docker check`: that command runs +-- `mx update-widgets` first, which reconciles the widget and reports 0 errors +-- while the stored value is still 0. Measured — the same project reads 0 errors +-- through `mxcli docker check` and 1 error through `mx check` (scripts/mx-check.sh). +-- +-- Then perturb: revert the fix (or stub the `stored` lookup in +-- hiddenUnnamedProperties) and re-run. The stored `maxHeight` goes back to 0 and +-- `mx check` reports the CE0463 above. + +create module ImgProbe; +/ + +-- The reported shape: an image with no dimensions given, so every dimension +-- property is left to the writer. +create or replace page ImgProbe.P_Image +( + title: 'One image', + layout: Atlas_Core.Atlas_Default +) +{ + container c1 { + image imgProbe (Image: 'Atlas_Core.Layout.logo', Responsive: false) + } +} +/ + +-- CONTROL: the dimensions MDL *can* name must survive. `maxHeightUnit` has no +-- MDL keyword — there is no authorable configuration in which `maxHeight` is +-- visible — so the risk this change carries is the opposite one: the reset now +-- consults the template, and a value the script gave must still outrank it. +-- Both `heightUnit`/`height` and `widthUnit`/`width` are mapped and named here, +-- and both must be stored as written rather than reset to 100/auto. +create or replace page ImgProbe.P_ImageSized +( + title: 'A sized image', + layout: Atlas_Core.Atlas_Default +) +{ + container c1 { + image imgSized ( + Image: 'Atlas_Core.Layout.logo', + Responsive: false, + WidthUnit: 'pixels', + Width: 200, + HeightUnit: 'pixels', + Height: 120 + ) + } +} +/ diff --git a/mdl-examples/bug-tests/ledger-27-consecutive-dynamictext.mdl b/mdl-examples/bug-tests/ledger-27-consecutive-dynamictext.mdl index 8b913e94b0..cb6850a43f 100644 --- a/mdl-examples/bug-tests/ledger-27-consecutive-dynamictext.mdl +++ b/mdl-examples/bug-tests/ledger-27-consecutive-dynamictext.mdl @@ -19,6 +19,20 @@ -- -- Usage (check passes; emits MDL-WIDGET15 info lines for the two fused pairs): -- mxcli check mdl-examples/bug-tests/ledger-27-consecutive-dynamictext.mdl +-- +-- CORRECTION (mendixlabs/mxcli#1046). The content params here used to be written +-- `{1} = $currentObject/Amount`. That is wrong, and this file demonstrated the +-- very class of defect it was filed under: a template parameter is evaluated +-- against the widget's own context object, so the `$currentObject/` prefix ends +-- up as part of the ATTRIBUTE NAME. Measured on 11.13.0 — the page as shipped +-- built with six errors: +-- +-- [CE1613] "The selected attribute 'MyModule.Summary.$currentObject/Amount' +-- no longer exists." at Text 'dMonth' (and five more) +-- +-- Nothing caught it because the example was only ever run through `mxcli check` +-- with no project and never executed. MDL-WIDGET24 now reports it with no +-- project at all, which is how this was found. -- ============================================================================ create entity MyModule.Summary ( Amount: Decimal, LastImport: DateTime ); @@ -33,18 +47,18 @@ create or replace page MyModule.SummaryPage dataview dv (datasource: $Summary) { -- Text-mode pair fuses → flagged container box { - dynamictext dMonth (content: 'This month: {1}', contentparams: [{1} = $currentObject/Amount]) - dynamictext dImport (content: 'Last import: {1}', contentparams: [{1} = $currentObject/LastImport]) + dynamictext dMonth (content: 'This month: {1}', contentparams: [{1} = Amount]) + dynamictext dImport (content: 'Last import: {1}', contentparams: [{1} = LastImport]) } -- Paragraph-mode pair ALSO fuses (renders inline) → flagged (#29) container paras { - dynamictext pMonth (content: 'This month: {1}', RenderMode: Paragraph, contentparams: [{1} = $currentObject/Amount]) - dynamictext pImport (content: 'Last import: {1}', RenderMode: Paragraph, contentparams: [{1} = $currentObject/LastImport]) + dynamictext pMonth (content: 'This month: {1}', RenderMode: Paragraph, contentparams: [{1} = Amount]) + dynamictext pImport (content: 'Last import: {1}', RenderMode: Paragraph, contentparams: [{1} = LastImport]) } -- Heading (block-level) + subtitle does NOT fuse → not flagged container heads { - dynamictext hTitle (content: 'Summary {1}', RenderMode: H2, contentparams: [{1} = $currentObject/Amount]) - dynamictext hSub (content: 'as of {1}', contentparams: [{1} = $currentObject/LastImport]) + dynamictext hTitle (content: 'Summary {1}', RenderMode: H2, contentparams: [{1} = Amount]) + dynamictext hSub (content: 'as of {1}', contentparams: [{1} = LastImport]) } } } diff --git a/mdl-examples/bug-tests/member-refs-resolved-at-check.mdl b/mdl-examples/bug-tests/member-refs-resolved-at-check.mdl new file mode 100644 index 0000000000..31195001c0 --- /dev/null +++ b/mdl-examples/bug-tests/member-refs-resolved-at-check.mdl @@ -0,0 +1,133 @@ +-- `check --references` resolved the ENTITY a create/change names and stopped +-- there, so a mistyped MEMBER passed check, passed exec, and surfaced at the far +-- end of a build (mendixlabs/mxcli#1048): +-- +-- CHANGE $Order ("IsArchived" = true); -- no such attribute +-- mxcli check -p app.mpr --references -> exit 0 +-- mxcli exec -> "Created microflow" +-- mx check -> [CE1613] "The selected attribute +-- 'Bench.Order.IsArchived' no longer exists." +-- +-- exec DOES resolve the name (resolveAttributeInEntityHierarchy) and, when +-- resolution fails, writes `.` anyway. That fabricated +-- identifier is what mxbuild rejects. +-- +-- Verify — everything below must PASS, which is the point: these are the shapes +-- that must not be reported. +-- +-- mxcli exec member-refs-resolved-at-check.mdl -p app.mpr +-- mxcli check member-refs-resolved-at-check.mdl -p app.mpr --references +-- mx check -p app.mpr -- 0 errors +-- +-- EXEC FIRST, and that ordering is the point rather than a convenience. The +-- check resolves members against the PROJECT, so against an app that has never +-- seen these entities every member is unknowable and the file passes having +-- exercised nothing. Measured: run the perturbation below against a virgin app +-- and it is silent; run it after the exec and it reports. A check that consults +-- the model can only speak about a model that exists. +-- +-- Then make ONE edit and re-check: change `"Status"` to `"Statuss"` in +-- ACT_Close. It is reported, with the entity's real members listed: +-- +-- MemberRefs.ACT_Close: MemberRefs.Order has no member "Statuss" +-- (in change $Order) — it has OrderNo, Status — mxbuild reports this as +-- CE1613 "The selected attribute 'MemberRefs.Order.Statuss' no longer exists" + +create module MemberRefs; +/ + +create or modify entity MemberRefs.Base ( + Code: String(20) +); +/ + +create or modify entity MemberRefs.Order extends MemberRefs.Base ( + OrderNo: String(50), + Status: String(50) +); +/ + +create or modify entity MemberRefs.Customer ( + Name: String(200) +); +/ + +create or modify association MemberRefs.Order_Customer + from MemberRefs.Order to MemberRefs.Customer type reference; +/ + +-- The entity of a CREATE is in the statement, so its members are always checked. +-- This one is the non-vacuity anchor: perturb any name here and it is reported. +create or modify microflow MemberRefs.ACT_New () +begin + $O = create MemberRefs.Order ( + "OrderNo" = 'A-1', + "Status" = 'Open' + ); + COMMIT $O; +end; +/ + +-- A CHANGE on a PARAMETER. Also covers the two shapes that would be false +-- positives if the walk were naive: +-- "Code" is INHERITED from MemberRefs.Base — the walk must follow +-- GeneralizationRef, or every specialisation reports its +-- inherited members as missing. +-- Order_Customer is an ASSOCIATION, which is a legal member of a change and +-- is not in the entity's attribute list. +create or modify microflow MemberRefs.ACT_Close ( + $Order: MemberRefs.Order, + $Customer: MemberRefs.Customer +) +begin + CHANGE $Order ( + "Status" = 'Closed', + "Code" = 'CLOSED', + Order_Customer = $Customer + ); + COMMIT $Order; +end; +/ + +-- A LOOP over a database retrieve: the iterator inherits the element type, so +-- the commonest bulk-update shape is checked rather than silently skipped. +create or modify microflow MemberRefs.ACT_CloseAll () +begin + RETRIEVE $Orders FROM MemberRefs.Order; + LOOP $O IN $Orders + BEGIN + CHANGE $O ("Status" = 'Closed'); + COMMIT $O; + END LOOP; +end; +/ + +-- A LOOP over an ASSOCIATION retrieve, traversed from the TO end. Resolving +-- this is what stops the check being silent on the shape real scripts use most. +create or modify microflow MemberRefs.ACT_CloseForCustomer ( + $Customer: MemberRefs.Customer +) +begin + RETRIEVE $Orders FROM $Customer/MemberRefs.Order_Customer; + LOOP $O IN $Orders + BEGIN + CHANGE $O ("OrderNo" = 'X'); + END LOOP; +end; +/ + +-- CONTROL: the migration shape. `Archived` does not exist in the stored project +-- when this script starts — the statement above adds it — so a check that only +-- consulted the project would report it and break the change most likely to +-- name a new member. +alter entity MemberRefs.Order add attribute "Archived": Boolean; +/ + +create or modify microflow MemberRefs.ACT_Archive ( + $Order: MemberRefs.Order +) +begin + CHANGE $Order ("Archived" = true); + COMMIT $Order; +end; +/ diff --git a/mdl-examples/bug-tests/messagedef-referenceset-cardinality.mdl b/mdl-examples/bug-tests/messagedef-referenceset-cardinality.mdl new file mode 100644 index 0000000000..98bfb66c13 --- /dev/null +++ b/mdl-examples/bug-tests/messagedef-referenceset-cardinality.mdl @@ -0,0 +1,135 @@ +-- A message definition's association cardinality was direction-only, and that +-- is the right rule for a Reference and the wrong one for a ReferenceSet +-- (ako/mxcli-rest FINDINGS #60). +-- +-- MaxOccurs says whether the exposed element is a single object or a list: +-- +-- forward (holder is FROM) reverse (holder is TO) +-- Reference 1 -1 +-- ReferenceSet -1 -1 +-- +-- mxcli returned 1 for every forward traversal, so a set was exposed as one +-- object. The direction rule was measured on a corpus in which all 927 +-- resolvable associations are `Reference` — it pins the direction half and says +-- nothing about the type half, which is why it looked exceptionless. +-- +-- Unlike the direction half, this one has a build error behind it. Measured on +-- ako/mxcli-rest's RestLab (Mendix 11.13.0), against a 0-error baseline, the +-- forward traversal of RateSnapshot_ExchangeRate produced: +-- +-- [error] [CE6524] "The domain model has changed and is no longer consistent +-- with the message definition … The occurrence of +-- 'RestLab.RateSnapshot_ExchangeRate' has changed" +-- at Entity message definition 'RateSnapshot' +-- [error] [CE0295] "Association 'RestLab.RateSnapshot_ExchangeRate' is not +-- allowed." at Object mapping element 'Rates' +-- +-- and 0 errors after the fix. CE6524's own advice ("resolve by refreshing the +-- message definition") is the tell: Mendix believes the domain model moved under +-- a definition that was in fact written wrong a second earlier. +-- +-- Verify: +-- mxcli exec messagedef-referenceset-cardinality.mdl -p app.mpr +-- mx check -p app.mpr -- 0 errors; before the fix, CE6524 + CE0295 +-- +-- The CONTROL is the Reference beside it: Order -> Customer must stay a single +-- object. A fix that returned -1 unconditionally clears CE6524 too. + +create module MsgCard; +/ + +create or modify entity MsgCard.Customer ( + Name: String(200) +); +/ + +create or modify entity MsgCard.Tag ( + Label: String(100) +); +/ + +create or modify entity MsgCard.Order ( + OrderNo: String(50) +); +/ + +-- The control: a plain Reference. One Order has one Customer, so reaching +-- Customer from Order is a single object (MaxOccurs 1). +create or modify association MsgCard.Order_Customer + from MsgCard.Order to MsgCard.Customer type reference; +/ + +-- The bug: a set is many in BOTH directions. +create or modify association MsgCard.Order_Tag + from MsgCard.Order to MsgCard.Tag type referenceset; +/ + +create or modify message definition collection MsgCard.MD_Orders +( + -- The forward traversal of each. Before the fix both stored MaxOccurs 1; + -- only Tags was wrong. + definition "OrderMsg" for MsgCard.Order as 'Order' ( + "OrderNo", + MsgCard.Order_Customer/MsgCard.Customer as 'Customer' ( + "Name" + ), + MsgCard.Order_Tag/MsgCard.Tag as 'Tags' ( + "Label" + ) + ), + -- The reverse of the set, which was already right and must stay right. + definition "TagMsg" for MsgCard.Tag as 'Tag' ( + "Label", + MsgCard.Order_Tag/MsgCard.Order as 'Orders' ( + "OrderNo" + ) + ) +); +/ + +-- The mapping is where CE0295 lands: an object mapping element bound to an +-- association whose occurrence disagrees with the domain model is refused. +create or modify export mapping MsgCard.EXM_Order + with message definition MsgCard.MD_Orders."OrderMsg" +{ + MsgCard.Order { + OrderNo = "OrderNo", + MsgCard.Order_Customer/MsgCard.Customer as Customer { + Name = "Name" + }, + MsgCard.Order_Tag/MsgCard.Tag as Tags { + Label = "Label" + } + } +}; +/ + +-- The second half of the same report: dropping an association a definition +-- still exposes was accepted silently, and `describe` went on emitting the +-- dangling member, so a describe -> exec round trip carried the break forward. +-- mxbuild is what caught it, as CE1613 at the definition. +-- +-- Not written as a statement here because the refusal is at exec time against a +-- project, which `mxcli check` (and so `make check-mdl`) does not reach. Run it +-- by hand after the statements above: +-- +-- $ mxcli -p app.mpr -c 'drop association MsgCard.Order_Tag' +-- Error: association MsgCard.Order_Tag is still exposed by a message +-- definition — dropping it would leave the definition bound to nothing +-- (CE1613). Remove the member first: +-- alter message definition MsgCard.MD_Orders.OrderMsg drop member Tag; +-- alter message definition MsgCard.MD_Orders.TagMsg drop member Order; +-- +-- BOTH are listed, because Order_Tag is exposed twice — once each way — and +-- clearing one still leaves the other dangling. The statements run verbatim, +-- and after both the drop succeeds. That is the whole point of printing them: +-- `drop member` matches the member's ORIGINAL name (the target entity's, `Tag`) +-- while an `in` path segment matches the EXPOSED one (`Tags`), so "remove the +-- member first" on its own sends the author to the name they wrote and a +-- "member not found". +-- +-- CONTROL: `drop association MsgCard.Order_Customer` after removing only the +-- Tags member must still be refused, and an association no definition names +-- (add one and drop it) must still drop. The guard is scoped to message +-- definitions alone — a microflow retrieve or an object mapping element over +-- the same association is still the author's own CE1613 to resolve. diff --git a/mdl-examples/bug-tests/navigation-icon-variants.mdl b/mdl-examples/bug-tests/navigation-icon-variants.mdl new file mode 100644 index 0000000000..5811a8d8e9 --- /dev/null +++ b/mdl-examples/bug-tests/navigation-icon-variants.mdl @@ -0,0 +1,65 @@ +-- ============================================================================ +-- Menu icons: all three of Mendix's icon elements round-trip +-- ============================================================================ +-- +-- Mendix stores THREE different icon elements on a menu item, and they are not +-- spellings of one value: +-- +-- Forms$IconCollectionIcon a qualified name in an icon collection +-- Forms$GlyphIcon a numeric character code, and no name at all +-- Forms$ImageIcon a qualified name in an IMAGE collection +-- +-- MDL could name only the first, so `describe navigation` emitted a comment for +-- the other two — and since CREATE NAVIGATION is a FULL REPLACEMENT, re-running +-- that output DELETED the icon the comment had just declined to describe. +-- +-- Measured on testdata/expr-checker, whose Home item carries a glyph icon: +-- +-- describe menu item 'Home' page …; +-- -- icon a numeric glyph code (Forms$GlyphIcon) is not reproducible … +-- exec Navigation profile 'Responsive' updated. +-- describe menu item 'Home' page …; <- the icon was destroyed +-- +-- Exit 0, success message, silent loss — the same shape as the pluggable-widget +-- body loss in mendixlabs/mxcli#1036. +-- +-- The bare form still means the icon-collection icon, so every script written +-- before this keeps its meaning. The keyword forms exist because writing a bare +-- name for an image icon would rebuild it as a COLLECTION icon: a silent variant +-- swap, and the reason the bare form was not simply widened to cover all three. + +create module NavIconVariants; + +create page NavIconVariants.Home ( Title: 'Home', Layout: Atlas_Core.Atlas_Default ) { + dynamictext t (Content: 'Home') +} + +create page NavIconVariants.Close ( Title: 'Close', Layout: Atlas_Core.Atlas_Default ) { + dynamictext t (Content: 'Close') +} + +create page NavIconVariants.Logo ( Title: 'Logo', Layout: Atlas_Core.Atlas_Default ) { + dynamictext t (Content: 'Logo') +} + +create or replace navigation Responsive + home page NavIconVariants.Home + menu ( + -- Forms$IconCollectionIcon — the bare form, unchanged + menu item 'Home' page NavIconVariants.Home icon Atlas_Core.Atlas.home; + -- Forms$GlyphIcon — the numeric character code IS the glyph's identity + menu item 'Close' page NavIconVariants.Close icon glyph 57377; + -- Forms$ImageIcon — an image collection, a different document + menu item 'Logo' page NavIconVariants.Logo icon image NavIconVariants.Images.logo; + -- A submenu takes any of the three too; it sits on the collapsed icon rail + menu 'Admin' icon glyph 57345 ( + menu item 'Users' page NavIconVariants.Home icon Atlas_Core.Atlas.user; + ); + ); + +-- The same item syntax serves a standalone menu document, which is why one AST +-- node backs both and the two cannot diverge. +create or modify menu NavIconVariants.Main_Menu ( + menu item 'Home' page NavIconVariants.Home icon Atlas_Core.Atlas.home; + menu item 'Close' page NavIconVariants.Close icon glyph 57377; +); diff --git a/mdl-examples/bug-tests/navigation-menu-item-icons.mdl b/mdl-examples/bug-tests/navigation-menu-item-icons.mdl new file mode 100644 index 0000000000..3d0995daa0 --- /dev/null +++ b/mdl-examples/bug-tests/navigation-menu-item-icons.mdl @@ -0,0 +1,50 @@ +-- ============================================================================ +-- MDL074: a navigation menu item with no icon is reported +-- ============================================================================ +-- +-- The navigation sidebar collapses to an icon rail, and that is the state most +-- users leave it in. A collapsed item shows its icon; one without falls back to +-- the first few characters of its caption, which is rarely enough to tell +-- "Orders" from "Order lines". Nothing caught it: the icon is optional in the +-- grammar, the model builds, and `mx check` passes. +-- +-- This script is a POSITIVE example — every item carries an icon, so it checks +-- clean. The warning it guards against is exercised by the unit tests in +-- mdl/executor/validate_navigation_icons_test.go, which can assert the absence +-- of a warning as well as its presence; a `.fail.mdl` cannot, because MDL074 is +-- a WARNING and `check` still exits 0. +-- +-- The rule needs no project, so it fires under `make check-mdl` too. + +create module NavIcons; + +create page NavIcons.Home ( Title: 'Home', Layout: Atlas_Core.Atlas_Default ) { + dynamictext t (Content: 'Home') +} + +create page NavIcons.Orders ( Title: 'Orders', Layout: Atlas_Core.Atlas_Default ) { + dynamictext t (Content: 'Orders') +} + +create page NavIcons.Users ( Title: 'Users', Layout: Atlas_Core.Atlas_Default ) { + dynamictext t (Content: 'Users') +} + +-- Every item iconed, including the SUBMENU PARENT — it sits directly on the +-- collapsed rail, so it needs one most of all. +create or replace navigation Responsive + home page NavIcons.Home + menu ( + menu item 'Home' page NavIcons.Home icon Atlas_Core.Atlas.home; + menu item 'Orders' page NavIcons.Orders icon Atlas_Core.Atlas."shopping-cart"; + menu 'Admin' icon Atlas_Core.Atlas.cog ( + menu item 'Users' page NavIcons.Users icon Atlas_Core.Atlas.user; + ); + ); + +-- A standalone menu document carries the same items and gets the same rule: +-- CREATE MENU and CREATE NAVIGATION share one AST node so they cannot diverge. +create or modify menu NavIcons.Main_Menu ( + menu item 'Home' page NavIcons.Home icon Atlas_Core.Atlas.home; + menu item 'Orders' page NavIcons.Orders icon Atlas_Core.Atlas."shopping-cart"; +); diff --git a/mdl-examples/bug-tests/update-security-runs-at-all.mdl b/mdl-examples/bug-tests/update-security-runs-at-all.mdl new file mode 100644 index 0000000000..4ddf56c994 --- /dev/null +++ b/mdl-examples/bug-tests/update-security-runs-at-all.mdl @@ -0,0 +1,67 @@ +-- `UPDATE SECURITY` was inert on every MPR v2 project (mendixlabs/mxcli#1047). +-- +-- It returned on the first module it could not reconcile, and one module always +-- can: SYSTEM. System's domain model is SYNTHESIZED rather than stored — there +-- is no unit file behind the id the module carries — so the reconcile could not +-- load it and the command died there: +-- +-- $ mxcli -p app.mpr -c 'update security' +-- Error: failed to reconcile security for module System: load domain model +-- 00000000-0000-0000-0000-000000000002: .../mprcontents/00/00/ +-- 00000000-0000-0000-0000-000000000002.mxunit: no such file or directory +-- +-- Reported as "UPDATE SECURITY does not fix the CE0066 it exists to fix". The +-- reason is that it never ran. Whether it had written some modules before dying +-- depended on where the unreconcilable one fell in the list, so it was also +-- non-atomic. +-- +-- A SECOND defect in the same command: `update security RestLab` — without IN — +-- reached the parser's error recovery, which consumed the module name silently. +-- The statement parsed as one statement with no error, `mxcli check` reported +-- "Syntax OK", and the run went PROJECT-WIDE. A scope the author asked for and +-- did not get is worse than a parse error. +-- +-- Verify (this script needs no fixture — it exercises the command itself): +-- +-- mxcli exec update-security-runs-at-all.mdl -p app.mpr +-- mx check -p app.mpr -- unchanged from before the run +-- +-- Every statement below must SUCCEED. Before the fix the first one failed with +-- the System error above, and the third silently ran project-wide. +-- +-- And two that must FAIL, run by hand because a script stops at the first error: +-- +-- $ mxcli -p app.mpr -c 'update security in System' +-- Error: System is the platform's module — its entity access rules are +-- Mendix's, not the project's, and its domain model is not stored in the .mpr +-- at all. There is nothing here to reconcile +-- +-- $ mxcli -p app.mpr -c 'update security NoSuchModule' +-- Error: module not found: NoSuchModule +-- +-- That second one is the reason a typo is refused rather than skipped: matching +-- no module used to reconcile nothing and print "All entity access rules are up +-- to date" — a success message for a run that did nothing. +-- +-- WHAT THIS EXAMPLE DOES NOT SHOW. It does not demonstrate a CE0066 being +-- repaired, and that is deliberate rather than an omission: mxcli reconciles an +-- entity's access rules on every write path, so it cannot produce a stale rule +-- to repair. Measured on 11.13.0 — `alter entity … add attribute` on both +-- engines, and a whole-entity `create or modify` rewrite, each left the project +-- at 0 errors. The original report's precondition ("adding an attribute leaves +-- one CE0066") therefore does not reproduce on current main. What is fixed here +-- is that the command RUNS; the repair itself is covered by unit tests, which +-- can construct the stale state this cannot. + +-- The project-wide run. This is the statement that used to fail. +update security; +/ + +-- Scoped with IN. +update security in Administration; +/ + +-- Scoped WITHOUT in — the same thing. This used to parse cleanly and then +-- reconcile every module in the project. +update security Administration; +/ diff --git a/mdl-examples/bug-tests/widget-member-refs-resolved-at-check.mdl b/mdl-examples/bug-tests/widget-member-refs-resolved-at-check.mdl new file mode 100644 index 0000000000..f28e93ff4e --- /dev/null +++ b/mdl-examples/bug-tests/widget-member-refs-resolved-at-check.mdl @@ -0,0 +1,134 @@ +-- Two member positions inside a widget that reference checking walked past. +-- Both passed `check --references`, passed `exec`, and failed the build. +-- +-- 1. An XPATH CONSTRAINT naming a member the entity does not have +-- (mendixlabs/mxcli#1049). The check resolved the entity in +-- `database from Bench.Order` and never looked inside the `where […]`: +-- +-- where [Bench.Order_Status = 'Open'] +-- mx check -> [CE1613] "The selected association 'Bench.Order_Status' +-- no longer exists." at Data grid 2 'gridOrders' +-- +-- 2. A CONTENTPARAMS value rooted in a VARIABLE (mendixlabs/mxcli#1046). A +-- template parameter is evaluated against the widget's own context object, +-- so `$Customer/` has nowhere to resolve and the writer keeps it as part of +-- the attribute name: +-- +-- ContentParams: [{1} = $Customer/Name] +-- mx check -> [CE1613] "The selected attribute +-- 'Bench.Customer.$Customer/Name' no longer exists." +-- +-- The two run in different TIERS, and the difference is the useful part: +-- +-- * the XPath one asks about the MODEL, so it needs -p; +-- * the ContentParams one is answerable from the statement alone, so it fires +-- under a bare `mxcli check` with no project (MDL-WIDGET24). +-- +-- That second tier is not theoretical. `ledger-27-consecutive-dynamictext.mdl` +-- had shipped in this directory using `{1} = $currentObject/Amount`, had only +-- ever been run through `mxcli check` with no project, and built with SIX +-- CE1613s the first time anyone executed it. MDL-WIDGET24 found it. +-- +-- Verify — EXEC FIRST, for the reason the sibling example gives: members are +-- resolved against the model, so against an app that has never seen these +-- entities every step is unknowable and the file passes having checked nothing. +-- +-- mxcli exec widget-member-refs-resolved-at-check.mdl -p app.mpr +-- mxcli check widget-member-refs-resolved-at-check.mdl -p app.mpr --references +-- mx check -p app.mpr -- 0 errors +-- +-- Then perturb, one at a time, and re-check: +-- +-- [Status = 'Open'] -> [Statuss = 'Open'] +-- WidgetRefs.P_List: datagrid "gridOrders": the constraint on +-- WidgetRefs.Order names "Statuss", which is neither an attribute nor an +-- association of it — … +-- +-- {1} = OrderNo -> {1} = $Order/OrderNo +-- (reported with NO project, by MDL-WIDGET24, naming `OrderNo` as the fix) +-- +-- The shapes MDL-WIDGET24 must NOT report, all measured on 11.13.0 by executing +-- and building each one: +-- +-- OrderNo clean +-- WidgetRefs.Order_Customer/Name clean (association hop) +-- $currentObject/WidgetRefs.Order_Customer/Name clean (prefix stripped) +-- $currentObject/OrderNo CE1613 +-- $Order/Name CE1613 + +create module WidgetRefs; +/ + +create or modify entity WidgetRefs.Customer ( + Name: String(200) +); +/ + +create or modify entity WidgetRefs.Order ( + OrderNo: String(50), + Status: String(50) +); +/ + +create or modify association WidgetRefs.Order_Customer + from WidgetRefs.Order to WidgetRefs.Customer type reference; +/ + +-- Every constraint here RESOLVES, which is the point: these are the shapes that +-- must not be reported. +create or replace page WidgetRefs.P_List +( + Title: 'Orders', + Layout: Atlas_Core.Atlas_Default +) +{ + -- A bare attribute of the constrained entity. + datagrid gridOrders (datasource: database from WidgetRefs.Order where [Status = 'Open']) { + column cNo (attribute: OrderNo, caption: 'No') + } + + -- An ASSOCIATION HOP and then an attribute of the entity it reaches. The walk + -- has to carry the entity from step to step, or this reads as a member of + -- Order and is wrongly reported. + datagrid gridByCustomer ( + datasource: database from WidgetRefs.Order + where [WidgetRefs.Order_Customer/WidgetRefs.Customer/Name = 'Acme'] + ) { + column cNo2 (attribute: OrderNo, caption: 'No') + } +} +/ + +-- ContentParams written the way the position wants: the attribute name, or an +-- association path to one. No `$variable/` root. +create or replace page WidgetRefs.P_Detail +( + params: { $Order: WidgetRefs.Order }, + title: 'Order', + layout: Atlas_Core.Atlas_Default +) +{ + dataview dvOrder (datasource: $Order) { + dynamictext dtNo (Content: 'Order {1}', ContentParams: [{1} = OrderNo]) + + -- An association path is LEGAL here — it is an attribute reached over an + -- association, stored as AttributeRef plus steps. Two segments, not three: + -- the association then the attribute. The XPath spelling + -- (`Assoc/Entity/Attr`) is NOT this, and mxbuild rejects it with the same + -- CE1613 — measured, and the reason this line is written out rather than + -- assumed. + dynamictext dtCust ( + Content: 'for {1}', + ContentParams: [{1} = WidgetRefs.Order_Customer/Name] + ) + + -- And `$currentObject/` IS stripped when an association path follows, so + -- this builds cleanly too. A check that flagged every variable root would + -- report it — the control that keeps MDL-WIDGET24 honest. + dynamictext dtCust2 ( + Content: 'also {1}', + ContentParams: [{1} = $currentObject/WidgetRefs.Order_Customer/Name] + ) + } +} +/ diff --git a/mdl-examples/doctype-tests/34-chart-widget-examples.mdl b/mdl-examples/doctype-tests/34-chart-widget-examples.mdl index 7a722d9974..ca630b2691 100644 --- a/mdl-examples/doctype-tests/34-chart-widget-examples.mdl +++ b/mdl-examples/doctype-tests/34-chart-widget-examples.mdl @@ -180,7 +180,7 @@ create page ChartExamples.P_Line ( StaticXAttribute: Period, StaticYAttribute: Total, StaticName: 'Revenue', - Interpolation: 'smooth' + Interpolation: 'spline' ) } } diff --git a/mdl-examples/doctype-tests/44-describe-widget-examples.mdl b/mdl-examples/doctype-tests/44-describe-widget-examples.mdl new file mode 100644 index 0000000000..9e97276c1a --- /dev/null +++ b/mdl-examples/doctype-tests/44-describe-widget-examples.mdl @@ -0,0 +1,43 @@ +-- DESCRIBE WIDGET — a widget definition, in-language. +-- +-- Slice 4 of PROPOSAL_def_driven_widget_bodies.md. A widget was the only MDL +-- extension point with no DESCRIBE: a microflow, nanoflow, Java action and +-- JavaScript action all describe in-language, against the live project. A +-- widget did not, which is WHY `mxcli widget init` generates markdown +-- documentation at all — and why that documentation could drift from what the +-- parser accepts (mendixlabs/mxcli#1036). +-- +-- The statement and `mxcli widget describe` are the same function +-- (executor.DescribeWidget), so they cannot disagree. + +-- By MDL keyword. +describe widget combobox; + +-- By full widget id — what a widget package, a page's BSON and the generated +-- docs all carry, and the only name a widget without a keyword has. +describe widget 'com.mendix.widget.web.htmlelement.HTMLElement'; + +-- Unlike every other DESCRIBE there is no qualified name: a widget definition +-- is not a document in the model. It comes from a package in the project, or +-- from mxcli's embedded set — which is also why this works with NO project +-- open, the state an agent is in when it asks "what can I write here?". +-- +-- The report covers each property's key, type, caption, category, whether it is +-- required, its default and its enumeration values, plus the dynamic rules the +-- widget's editor uses to hide properties under some configurations. Those +-- rules matter: writing into a hidden property is the CE0463 that +-- mdl-examples/bug-tests/956-fileuploader-six-action-slots.mdl documents. + +-- The report ends with an MDL example that PARSES AS WRITTEN. Both the head +-- form and the containers in it are chosen by probing the real parser, so: +-- +-- gallery -> `gallery widget1 (…) { filter slot1 {…} template slot2 {…} }` +-- its `emptyplaceholder` slot is omitted and named, because +-- the grammar has no keyword for it yet +-- htmlelement -> the `pluggablewidget '' …` form, because `htmlelement` +-- is not yet a widget keyword, and all four of its containers +-- are omitted and named +-- +-- This is the half of the generated .md that was wrong — its example failed on +-- its own first line. Deriving it means it cannot promise syntax that fails, +-- and that it widens on its own once slices 2-3 land. diff --git a/mdl/ast/ast_enumeration.go b/mdl/ast/ast_enumeration.go index 0445342ad8..e46c780a5b 100644 --- a/mdl/ast/ast_enumeration.go +++ b/mdl/ast/ast_enumeration.go @@ -67,6 +67,13 @@ type AlterEnumerationStmt struct { ValueName string NewName string // For RENAME Caption string // For ADD and MODIFY CAPTION + + // Idempotency guards, so a script that adds an enumeration value is + // re-runnable. Without them the second run errors and exec STOPS THERE, + // leaving every later statement unapplied. Same pair as ALTER ENTITY's + // ADD ATTRIBUTE / DROP INDEX. (ako/mxcli-rest FINDINGS #60) + IfNotExists bool // ADD VALUE IF NOT EXISTS + IfExists bool // DROP VALUE IF EXISTS } func (s *AlterEnumerationStmt) isStatement() {} diff --git a/mdl/ast/ast_navigation.go b/mdl/ast/ast_navigation.go index af81e8f317..f2bc41c883 100644 --- a/mdl/ast/ast_navigation.go +++ b/mdl/ast/ast_navigation.go @@ -2,6 +2,8 @@ package ast +import "github.com/mendixlabs/mxcli/mdl/types" + // AlterNavigationStmt represents: CREATE [OR REPLACE] NAVIGATION [clauses...] // This is a full-replacement command: omitted clauses clear that section. type AlterNavigationStmt struct { @@ -25,12 +27,19 @@ type NavHomePageDef struct { // NavMenuItemDef represents a MENU ITEM or MENU sub-menu definition. type NavMenuItemDef struct { - Caption string // from STRING_LITERAL - Page *QualifiedName // PAGE target - Microflow *QualifiedName // MICROFLOW target - SignOut bool // SIGN_OUT — the third action a menu item can carry - Icon string // ICON 'Module.Collection.name', empty for none - Items []NavMenuItemDef // Sub-items (for MENU 'caption' (...)) + Caption string // from STRING_LITERAL + Page *QualifiedName // PAGE target + Microflow *QualifiedName // MICROFLOW target + SignOut bool // SIGN_OUT — the third action a menu item can carry + Icon string // the qualified name, for the collection and image kinds + // IconKind says WHICH of Mendix's three icon elements was written. They are + // not variants of one value — a glyph carries a numeric code and no name — + // so a single Icon string could express only one of the three, and the other + // two were destroyed on rewrite. + IconKind types.MenuIconKind + // IconCode is the glyph's numeric character code, set only for MenuIconGlyph. + IconCode int + Items []NavMenuItemDef // Sub-items (for MENU 'caption' (...)) } // CreateMenuStmt is `create [or modify] menu Module.Name ( )` — a diff --git a/mdl/ast/ast_page_v3.go b/mdl/ast/ast_page_v3.go index 5ee5ff6129..923329d5ce 100644 --- a/mdl/ast/ast_page_v3.go +++ b/mdl/ast/ast_page_v3.go @@ -107,6 +107,23 @@ type WidgetV3 struct { // no name, and carries nothing but this entity and its widgets. Empty for // every other widget, including a Gallery's named `template ` slot. Specialization string + + // TypeIsGeneric records that Type came from the grammar's generic + // IDENTIFIER alternative rather than one of the enumerated widget-type + // tokens (slice 2 of PROPOSAL_def_driven_widget_bodies.md). + // + // The distinction is invisible in Type — both arrive as a lowercase string — + // but it is what tells a typo from a built-in. `htmlelemnt` can ONLY be a + // misspelt widget definition, because a real built-in has its own token; a + // generic type that resolves to no definition is therefore MDL-WIDGET25 + // rather than a static widget to be validated on the builtin property + // vocabulary. Without it the typo passes `check` with a warning about the + // wrong thing. + // + // Set by the visitor from the parse tree, never inferred from a list of + // known widget names — inferring it would reintroduce the list this + // proposal exists to remove. + TypeIsGeneric bool } // DataSourceV3 represents a V3 datasource expression. diff --git a/mdl/ast/ast_query.go b/mdl/ast/ast_query.go index 82cd695249..ee115cf89c 100644 --- a/mdl/ast/ast_query.go +++ b/mdl/ast/ast_query.go @@ -348,8 +348,11 @@ const ( DescribeMenu // DESCRIBE MENU Module.Name (standalone Menus$MenuDocument) DescribeQueue // DESCRIBE QUEUE Module.Name DescribeScheduledEvent // DESCRIBE SCHEDULED EVENT Module.Name - DescribeRegularExpression // DESCRIBE REGULAR EXPRESSION Module.Name - DescribeAuto // DESCRIBE Module.Name — type auto-detected at execution time + // DescribeWidget is not a document — it is a widget DEFINITION, named by + // MDL keyword or widget id. Name carries whichever was written. + DescribeWidget // DESCRIBE WIDGET combobox | DESCRIBE WIDGET 'com.mendix.widget.web.combobox.Combobox' + DescribeRegularExpression // DESCRIBE REGULAR EXPRESSION Module.Name + DescribeAuto // DESCRIBE Module.Name — type auto-detected at execution time ) // String returns the human-readable name of the describe object type. diff --git a/mdl/backend/mcp/widget.go b/mdl/backend/mcp/widget.go index 5f6bc8ca70..b6129aff50 100644 --- a/mdl/backend/mcp/widget.go +++ b/mdl/backend/mcp/widget.go @@ -542,6 +542,12 @@ func (w *mcpWidgetBuilder) PropertyTypeIDs() map[string]pages.PropertyTypeIDEntr return out } +// PrimitiveValues has nothing to report: this backend records property SETS and +// hands them to Studio Pro, which expands every default itself, so there is no +// template object here to read a captured value out of. Empty is the honest +// answer and leaves the caller on its declared-default fallback. +func (w *mcpWidgetBuilder) PrimitiveValues() map[string]string { return nil } + func (w *mcpWidgetBuilder) EnsureRequiredObjectLists() {} func (w *mcpWidgetBuilder) ApplyPropertyVisibility(_ []types.WidgetVisibilityRule) {} diff --git a/mdl/backend/modelsdk/menu_write.go b/mdl/backend/modelsdk/menu_write.go index 0f6e4869e1..dde62538e1 100644 --- a/mdl/backend/modelsdk/menu_write.go +++ b/mdl/backend/modelsdk/menu_write.go @@ -120,19 +120,56 @@ func menuCaptionToGen(caption string) element.Element { return t } -// menuIconToGen emits only Forms$IconCollectionIcon, the one variant MDL can -// name. A glyph icon carries a numeric code and an image icon points into an -// image collection; neither is expressible in the ICON clause, so an item that -// had one keeps no icon rather than getting a wrong one. DESCRIBE flags those on -// the way out, so the loss is visible rather than silent. +// menuIconToGen emits whichever of Mendix's three icon elements the item +// carries. +// +// It used to emit only Forms$IconCollectionIcon, because that was the one +// variant the ICON clause could name — so an item holding a glyph or image icon +// was rewritten with NO icon. `create or modify menu` is a full replacement, so +// that was deletion, not omission: the item came back without the icon it went +// in with, at exit 0. +// +// Nothing is inferred here. The kind comes from the author's `icon glyph …` / +// `icon image …` or from the kind the reader saw in storage. func menuIconToGen(item *types.NavMenuItem) element.Element { - if item.Icon == "" { - return nil + kind := types.MenuIconKindOf(item.IconType) + if kind == types.MenuIconNone && item.Icon != "" { + // An item built before the kind existed carries a name and nothing else, + // and that name has only ever meant an icon-collection icon. + kind = types.MenuIconCollection + } + switch kind { + case types.MenuIconGlyph: + // A glyph with no code identifies no glyph; an empty element is worse + // than none, because it renders as a blank where an icon should be. + if item.IconCode == 0 { + return nil + } + icon := genPages.NewGlyphIcon() + icon.SetID(element.ID(mmpr.GenerateID())) + icon.SetCode(int32(item.IconCode)) + return icon + case types.MenuIconImage: + if item.Icon == "" { + return nil + } + icon := genPages.NewImageIcon() + icon.SetID(element.ID(mmpr.GenerateID())) + icon.SetImageQualifiedName(item.Icon) + return icon + case types.MenuIconCollection: + if item.Icon == "" { + return nil + } + icon := genPages.NewIconCollectionIcon() + icon.SetID(element.ID(mmpr.GenerateID())) + icon.SetImageQualifiedName(item.Icon) + return icon } - icon := genPages.NewIconCollectionIcon() - icon.SetID(element.ID(mmpr.GenerateID())) - icon.SetImageQualifiedName(item.Icon) - return icon + // MenuIconUnknown: a stored $Type this build does not know. Emitting nothing + // would delete it, so the codec must refuse rather than guess — see the + // caller, which turns this into an error. + return nil } // menuActionToGen builds the item's client action. The gen type names are the SDK diff --git a/mdl/backend/modelsdk/navigation_icon_test.go b/mdl/backend/modelsdk/navigation_icon_test.go index 4e796ae9c6..37c7ad7b12 100644 --- a/mdl/backend/modelsdk/navigation_icon_test.go +++ b/mdl/backend/modelsdk/navigation_icon_test.go @@ -25,7 +25,7 @@ func navIconEntry(d bson.D, key string) (interface{}, bool) { // divergence here means MXCLI_ENGINE silently changes what lands in the .mpr — // and only one of the two would open in Studio Pro. func TestNavMenuIconBson_MatchesTheMprEngine(t *testing.T) { - d, ok := navMenuIconBson("Atlas_Core.Atlas.align-center").(bson.D) + d, ok := navMenuIconBson(types.NavMenuItemSpec{Icon: "Atlas_Core.Atlas.align-center"}).(bson.D) if !ok { t.Fatal("expected a bson.D") } @@ -41,8 +41,8 @@ func TestNavMenuIconBson_MatchesTheMprEngine(t *testing.T) { } func TestNavMenuIconBson_EmptyNameStaysNull(t *testing.T) { - if got := navMenuIconBson(""); got != nil { - t.Errorf("navMenuIconBson(\"\") = %v, want nil", got) + if got := navMenuIconBson(types.NavMenuItemSpec{}); got != nil { + t.Errorf("navMenuIconBson(empty spec) = %v, want nil", got) } } @@ -78,9 +78,9 @@ func TestNavFormSettingsBson_NoTitleOverrideStaysNull(t *testing.T) { } func TestMenuIconOf_NilIconYieldsNothing(t *testing.T) { - typeName, image := menuIconOf(nil) - if typeName != "" || image != "" { - t.Errorf("menuIconOf(nil) = (%q, %q), want empty", typeName, image) + typeName, image, code := menuIconOf(nil) + if typeName != "" || image != "" || code != 0 { + t.Errorf("menuIconOf(nil) = (%q, %q, %d), want empty", typeName, image, code) } } @@ -116,7 +116,7 @@ func TestMenuIconOf_ReadsTheNameOffARegisteredVariant(t *testing.T) { } { t.Run(tc.typeName, func(t *testing.T) { el := decodeIcon(t, tc.typeName, bsonv2.E{Key: "Image", Value: tc.image}) - gotType, gotImage := menuIconOf(el) + gotType, gotImage, _ := menuIconOf(el) if gotType != tc.typeName { t.Errorf("type = %q, want %q", gotType, tc.typeName) } @@ -131,7 +131,7 @@ func TestMenuIconOf_ReadsTheNameOffARegisteredVariant(t *testing.T) { // DESCRIBE from emitting a lossy ICON clause for it. func TestMenuIconOf_GlyphHasNoName(t *testing.T) { el := decodeIcon(t, "Forms$GlyphIcon", bsonv2.E{Key: "Code", Value: int32(9999)}) - gotType, gotImage := menuIconOf(el) + gotType, gotImage, _ := menuIconOf(el) if gotType != "Forms$GlyphIcon" { t.Errorf("type = %q", gotType) } @@ -139,3 +139,33 @@ func TestMenuIconOf_GlyphHasNoName(t *testing.T) { t.Errorf("image = %q, want empty: a glyph has no qualified name", gotImage) } } + +// The glyph's Code is the only thing that says WHICH glyph. Reading the $Type +// alone left a caller knowing one was there and nothing more, so DESCRIBE could +// not re-emit it and a rewrite replaced it with nothing. +func TestMenuIconOf_ReadsTheGlyphCode(t *testing.T) { + el := decodeIcon(t, "Forms$GlyphIcon", bsonv2.E{Key: "Code", Value: int32(57345)}) + gotType, gotImage, gotCode := menuIconOf(el) + if gotType != "Forms$GlyphIcon" { + t.Errorf("type = %q, want Forms$GlyphIcon", gotType) + } + if gotImage != "" { + t.Errorf("image = %q, want empty — a glyph carries no qualified name", gotImage) + } + if gotCode != 57345 { + t.Errorf("code = %d, want 57345", gotCode) + } +} + +// The control: a collection icon must NOT pick up a code, or "has a code" stops +// distinguishing a glyph from anything else. +func TestMenuIconOf_CollectionIconHasNoCode(t *testing.T) { + el := decodeIcon(t, "Forms$IconCollectionIcon", bsonv2.E{Key: "Image", Value: "Atlas_Core.Atlas.home"}) + _, gotImage, gotCode := menuIconOf(el) + if gotImage != "Atlas_Core.Atlas.home" { + t.Errorf("image = %q", gotImage) + } + if gotCode != 0 { + t.Errorf("code = %d, want 0", gotCode) + } +} diff --git a/mdl/backend/modelsdk/navigation_read.go b/mdl/backend/modelsdk/navigation_read.go index 7b98df53d5..9e4ee77662 100644 --- a/mdl/backend/modelsdk/navigation_read.go +++ b/mdl/backend/modelsdk/navigation_read.go @@ -237,7 +237,7 @@ func navMenuItemFromGen(el element.Element) *types.NavMenuItem { item := &types.NavMenuItem{ Caption: textOf(mi.Caption()), } - item.IconType, item.Icon = menuIconOf(mi.Icon()) + item.IconType, item.Icon, item.IconCode = menuIconOf(mi.Icon()) resolveMenuAction(item, mi.Action()) for _, subEl := range mi.ItemsItems() { if sub := navMenuItemFromGen(subEl); sub != nil { @@ -262,21 +262,31 @@ func navMenuItemFromGen(el element.Element) *types.NavMenuItem { // nothing for exactly the registered variants — which is what made DESCRIBE drop // every Atlas icon under this engine while the legacy engine printed them. Match // on the Raw() method instead, which both shapes satisfy. -func menuIconOf(icon element.Element) (typeName, image string) { +func menuIconOf(icon element.Element) (typeName, image string, code int) { if icon == nil { - return "", "" + return "", "", 0 } typeName = icon.TypeName() raw, ok := icon.(interface{ Raw() bson.Raw }) if !ok { - return typeName, "" + return typeName, "", 0 } if v, err := raw.Raw().LookupErr("Image"); err == nil { if s, ok := v.StringValueOK(); ok { image = s } } - return typeName, image + // Forms$GlyphIcon's Code is the ONLY thing identifying which glyph it is. + // Reading the $Type alone told a caller a glyph was there and nothing more, + // so DESCRIBE could not re-emit it and a rewrite replaced it with nothing. + if v, err := raw.Raw().LookupErr("Code"); err == nil { + if i, ok := v.Int32OK(); ok { + code = int(i) + } else if i, ok := v.Int64OK(); ok { + code = int(i) + } + } + return typeName, image, code } // resolveMenuAction sets the action type / target on a NavMenuItem from a gen diff --git a/mdl/backend/modelsdk/navigation_write.go b/mdl/backend/modelsdk/navigation_write.go index 1fd184d02e..58ce1bbe21 100644 --- a/mdl/backend/modelsdk/navigation_write.go +++ b/mdl/backend/modelsdk/navigation_write.go @@ -278,7 +278,7 @@ func navMenuItemBson(mi types.NavMenuItemSpec) bson.D { {Key: "Action", Value: navMenuAction(mi)}, {Key: "AlternativeText", Value: nil}, {Key: "Caption", Value: navCaptionBson(mi.Caption)}, - {Key: "Icon", Value: navMenuIconBson(mi.Icon)}, + {Key: "Icon", Value: navMenuIconBson(mi)}, } subItems := bson.A{navMarkerItems} for _, sub := range mi.Items { @@ -288,18 +288,42 @@ func navMenuItemBson(mi types.NavMenuItemSpec) bson.D { return item } -// navMenuIconBson mirrors sdk/mpr's buildMenuIconBson: the storage name is -// Forms$IconCollectionIcon (not the metamodel's Pages$…), and only that variant -// is emitted. See the comment there for why GlyphIcon/ImageIcon are excluded. -func navMenuIconBson(icon string) interface{} { - if icon == "" { +// navMenuIconBson mirrors sdk/mpr's buildMenuIconBson. The storage names are +// Forms$… (not the metamodel's Pages$…) — "Form" was the original term for +// "Page". +// +// All THREE variants are emitted. Only the icon-collection one used to be, so a +// glyph or image icon read off a real project came back as no icon at all, and +// `create or replace navigation` — a full replacement — wrote that nothing over +// the user's icon. Measured on testdata/expr-checker: exec of DESCRIBE's own +// output destroyed the Home item's glyph icon at exit 0. +func navMenuIconBson(spec types.NavMenuItemSpec) interface{} { + kind := spec.IconKind + if kind == types.MenuIconNone && spec.Icon != "" { + // A spec built before the kind existed carries a name and nothing else. + // That name has only ever meant an icon-collection icon. + kind = types.MenuIconCollection + } + storage := types.MenuIconStorageType(kind) + if storage == "" { return nil } - return bson.D{ + doc := bson.D{ {Key: "$ID", Value: navID()}, - {Key: "$Type", Value: "Forms$IconCollectionIcon"}, - {Key: "Image", Value: icon}, + {Key: "$Type", Value: storage}, + } + if kind == types.MenuIconGlyph { + // A glyph with no code identifies no glyph; writing one would store an + // icon nobody can see. Emit no icon rather than an empty element. + if spec.IconCode == 0 { + return nil + } + return append(doc, bson.E{Key: "Code", Value: int32(spec.IconCode)}) + } + if spec.Icon == "" { + return nil } + return append(doc, bson.E{Key: "Image", Value: spec.Icon}) } func navCaptionBson(text string) bson.D { diff --git a/mdl/backend/mpr/convert.go b/mdl/backend/mpr/convert.go index 658a82c7de..63bf8acadf 100644 --- a/mdl/backend/mpr/convert.go +++ b/mdl/backend/mpr/convert.go @@ -234,7 +234,7 @@ func convertNavProfile(in *mpr.NavigationProfile) *types.NavigationProfile { func convertNavMenuItem(in *mpr.NavMenuItem) *types.NavMenuItem { mi := &types.NavMenuItem{ Caption: in.Caption, Page: in.Page, Microflow: in.Microflow, ActionType: in.ActionType, - Icon: in.Icon, IconType: in.IconType, + Icon: in.Icon, IconType: in.IconType, IconCode: in.IconCode, } if in.Items != nil { mi.Items = make([]*types.NavMenuItem, len(in.Items)) diff --git a/mdl/backend/mutation.go b/mdl/backend/mutation.go index 6f49aed60f..ddc7ae8c33 100644 --- a/mdl/backend/mutation.go +++ b/mdl/backend/mutation.go @@ -367,6 +367,12 @@ type WidgetObjectBuilder interface { // PropertyTypeIDs returns the property type metadata for the loaded template. PropertyTypeIDs() map[string]pages.PropertyTypeIDEntry + // PrimitiveValues returns each property's current comparable value in the + // object being built. Read before the mappings are applied it is the + // template's captured configuration, which is what a property MDL cannot + // name will actually be stored with. + PrimitiveValues() map[string]string + // --- Object list defaults --- // EnsureRequiredObjectLists auto-populates required empty object lists. diff --git a/mdl/backend/widgetobj/builder.go b/mdl/backend/widgetobj/builder.go index f01c85209b..fc985e3401 100644 --- a/mdl/backend/widgetobj/builder.go +++ b/mdl/backend/widgetobj/builder.go @@ -721,6 +721,18 @@ func (ob *Builder) PropertyTypeIDs() map[string]pages.PropertyTypeIDEntry { return ob.propertyTypeIDs } +// PrimitiveValues returns each property's current comparable value in the object +// being built, keyed as the widget declares it. +// +// Read before the property mappings are applied, this is the TEMPLATE's captured +// configuration — which is what an UNMAPPED property will actually be stored +// with, since nothing else ever touches one. That distinction is what a +// visibility rule has to be evaluated against: the declared default is what the +// property SHOULD hold, not what it will (see hiddenUnnamedProperties). +func (ob *Builder) PrimitiveValues() map[string]string { + return primitiveValuesOf(ob.object, ob.propertyTypeIDs) +} + // --------------------------------------------------------------------------- // Object list defaults // --------------------------------------------------------------------------- diff --git a/mdl/catalog/builder_pages.go b/mdl/catalog/builder_pages.go index 555feb57ac..ef9ed48daa 100644 --- a/mdl/catalog/builder_pages.go +++ b/mdl/catalog/builder_pages.go @@ -554,21 +554,9 @@ func extractWidgetsRecursive(w map[string]any) []rawWidgetInfo { } } - // Handle CustomWidget nested widgets in properties + // Handle CustomWidget nested widgets in properties — both kinds of container. if obj, ok := w["Object"].(map[string]any); ok { - props := getBsonArrayElements(obj["Properties"]) - for _, prop := range props { - if propMap, ok := prop.(map[string]any); ok { - if value, ok := propMap["Value"].(map[string]any); ok { - propWidgets := getBsonArrayElements(value["Widgets"]) - for _, pw := range propWidgets { - if pwMap, ok := pw.(map[string]any); ok { - result = append(result, extractWidgetsRecursive(pwMap)...) - } - } - } - } - } + result = append(result, widgetsInPropertyBag(obj)...) } // Handle NavigationList items @@ -587,6 +575,51 @@ func extractWidgetsRecursive(w map[string]any) []rawWidgetInfo { return result } +// widgetsInPropertyBag walks a pluggable widget's stored property bag and +// indexes every widget inside it, through BOTH kinds of container: +// +// Value.Widgets a child slot — a Gallery's content, an HTML Element's body +// Value.Objects an object list — a DataGrid2 column, a chart series +// +// Only the first was walked, so anything placed in a column, a gallery item or +// a series was invisible to the catalog. Measured on a real project: a page +// holding 19 chart sparklines inside datagrid columns did not appear under +// "which pages use VegaChart?", while the grid around them did. +// +// The consequence is wider than the widget edge, because CATALOG.REFS is a +// projection of this table: an entity or microflow used ONLY inside a column +// template reported zero references, so anything using reference counts to +// decide "unused, safe to delete" would delete a document in active use. That is +// issue #940's failure mode — fixed for List View templates, left open here. +// +// An object-list item is itself a property bag, so the walk recurses: a column +// holding a nested widget that has its own object list is covered without a +// second case. +func widgetsInPropertyBag(bag map[string]any) []rawWidgetInfo { + var result []rawWidgetInfo + for _, prop := range getBsonArrayElements(bag["Properties"]) { + propMap, ok := prop.(map[string]any) + if !ok { + continue + } + value, ok := propMap["Value"].(map[string]any) + if !ok { + continue + } + for _, pw := range getBsonArrayElements(value["Widgets"]) { + if pwMap, ok := pw.(map[string]any); ok { + result = append(result, extractWidgetsRecursive(pwMap)...) + } + } + for _, obj := range getBsonArrayElements(value["Objects"]) { + if objMap, ok := obj.(map[string]any); ok { + result = append(result, widgetsInPropertyBag(objMap)...) + } + } + } + return result +} + // extractSnippetWidgets extracts all widgets from raw snippet BSON data. func extractSnippetWidgets(rawData map[string]any) []rawWidgetInfo { // Handle both snippet formats: diff --git a/mdl/catalog/builder_pages_objectlist_test.go b/mdl/catalog/builder_pages_objectlist_test.go new file mode 100644 index 0000000000..9709365a5e --- /dev/null +++ b/mdl/catalog/builder_pages_objectlist_test.go @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import "testing" + +// gridWithWidgetInAColumn is the BSON shape of a pluggable widget whose +// object-list ITEM holds widgets: a DataGrid2 column with custom content, a +// gallery item, a chart series. The item lives in `Value.Objects`, and each +// object carries its own `Properties[].Value.Widgets`. +func gridWithWidgetInAColumn() map[string]any { + return map[string]any{ + "$ID": "grid-1", + "Name": "grid1", + "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{"WidgetId": "com.mendix.widget.web.datagrid.Datagrid"}, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t-columns", + "Value": map[string]any{ + "Objects": []any{ + int32(3), + map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t-content", + "Value": map[string]any{ + "Widgets": []any{ + int32(3), + map[string]any{ + "$ID": "pb-1", + "Name": "pb1", + "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{ + "WidgetId": "com.mendix.widget.custom.progressbar.ProgressBar", + }, + }, + }, + }, + }, + }, + }, + }, + }, + }, + }, + }, + } +} + +// A widget inside an object-list item must be indexed. +// +// The catalog walked a pluggable widget's `Object.Properties[].Value.Widgets` +// (a child slot) but not `Value.Objects[]` (an object list), so anything placed +// in a DataGrid2 column, a gallery item or a chart series was invisible. +// +// Measured on a real project by an external test: a page holding 19 chart +// sparklines inside datagrid columns did not appear under "which pages use +// VegaChart?", while the grid around them did. Reproduced here on the fixture — +// a progressbar in a `column c2 { … }` gave one row (the grid), ground truth two. +// +// It matters beyond the widget edge: CATALOG.REFS is a projection of this table, +// so an entity or microflow used ONLY inside a column template reported zero +// references, and anything using reference counts to decide "unused, safe to +// delete" would delete a document in active use. That is issue #940's failure +// mode, which was fixed for List View templates and left open here. +func TestExtractWidgetsRecursive_ObjectListItemWidgets(t *testing.T) { + got := extractWidgetsRecursive(gridWithWidgetInAColumn()) + + var sawGrid, sawNested bool + for _, w := range got { + switch w.WidgetType { + case "com.mendix.widget.web.datagrid.Datagrid": + sawGrid = true + case "com.mendix.widget.custom.progressbar.ProgressBar": + sawNested = true + } + } + if !sawGrid { + t.Error("the grid itself was not indexed") + } + if !sawNested { + t.Errorf("the widget inside the column was not indexed; got %d widgets: %+v", + len(got), got) + } +} + +// The control: a child slot (Value.Widgets, no Objects) must keep working. A fix +// that swapped one traversal for the other would pass the test above and lose +// every Gallery content widget. +func TestExtractWidgetsRecursive_ChildSlotStillWalked(t *testing.T) { + w := map[string]any{ + "$ID": "g-1", "Name": "gal1", "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{"WidgetId": "com.mendix.widget.web.gallery.Gallery"}, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t-content", + "Value": map[string]any{ + "Widgets": []any{ + int32(3), + map[string]any{"$ID": "t-1", "Name": "txt1", "$Type": "Forms$DynamicText"}, + }, + }, + }, + }, + }, + } + var sawText bool + for _, got := range extractWidgetsRecursive(w) { + if got.WidgetType == "Forms$DynamicText" { + sawText = true + } + } + if !sawText { + t.Error("a widget in a child slot stopped being indexed") + } +} + +// The second control: an object list with no widgets in it must not invent rows, +// and must not panic on the marker-only array. getBsonArrayElements strips the +// leading typed-array marker, so an empty list is length 0 here and length 1 raw. +func TestExtractWidgetsRecursive_EmptyObjectList(t *testing.T) { + w := map[string]any{ + "$ID": "g-1", "Name": "c1", "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{"WidgetId": "com.acme.Thing"}, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t", + "Value": map[string]any{"Objects": []any{int32(3)}}, + }, + }, + }, + } + if got := extractWidgetsRecursive(w); len(got) != 1 { + t.Errorf("got %d widgets, want 1 (the widget itself): %+v", len(got), got) + } +} diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index d516e88c20..2e32c485aa 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -34,6 +34,7 @@ const ( RefKindReturn = "return" // Microflow/nanoflow returns an entity type RefKindSchedule = "schedule" // Scheduled event runs a microflow RefKindValidate = "validate" // Attribute validation rule uses a regular expression + RefKindWidget = "widget" // Page/snippet uses a pluggable or custom widget ) // collectActionActivities returns all ActionActivity objects from an ObjectCollection, @@ -426,6 +427,12 @@ func (b *Builder) buildReferences() error { } } } + + // Page/snippet -> widget definition. buildWidgetDefinitions runs before + // this pass, so the join target is populated. + if n, werr := insertWidgetRefs(b.tx, projectID, snapshotID); werr == nil { + refCount += n + } } // Extract navigation references (home pages, menu items, login pages) diff --git a/mdl/catalog/builder_widget_refs.go b/mdl/catalog/builder_widget_refs.go new file mode 100644 index 0000000000..8ef654ee24 --- /dev/null +++ b/mdl/catalog/builder_widget_refs.go @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +// insertWidgetRefs emits the `widget` edge: one row per (page or snippet) x +// widget definition actually used on it. +// +// Slice 5 of PROPOSAL_def_driven_widget_bodies.md. A widget was the one MDL +// extension point with no edge in CATALOG.REFS, so "which pages use this +// widget?" was unanswerable while the same question about a Java action was one +// query. It is also the question an upgrade asks: a .mpk shipped in widgets/ +// that no page uses is dead weight, and one used on forty pages is not +// something to swap lightly. +// +// Both halves are already in the catalog, so this is a projection and costs no +// extra parse: widgets_data.WidgetType carries the widget ID for a pluggable or +// custom widget (buildPages resolves Type.WidgetId out of the +// CustomWidgets$CustomWidget wrapper), and widget_definitions_data is keyed by +// that same ID. +// +// # Only widgets that resolve to a definition +// +// The join is the filter. A built-in Mendix widget stores its BSON $Type in the +// same column (Forms$DynamicText, Forms$ActionButton, ...) and has no +// definition, so it gets no edge — deliberately. An edge is a pointer to +// something describable, and `Forms$TextBox` is a language primitive, not a +// document: emitting one would put a target in the graph that nothing can +// resolve. "Which pages have a text box?" is already answerable directly from +// CATALOG.WIDGETS. +// +// # TargetName is the MDL name, not the widget ID +// +// Measured on testdata/expr-checker (15 widget edges either way), with the +// widget ID as TargetName: +// +// graph_module_coupling gains a module "com" with 14 edges, from three +// different source modules +// graph_god_nodes reports com.mendix.widget.web.image.Image with +// ModuleName "com" +// +// Those views derive a module by taking everything before the FIRST dot, which +// is sound for a qualified name and nonsense for a dotted widget ID. Using the +// MDL name (IMAGE, COMBOBOX) leaves graph_module_coupling identical to the +// baseline, because its existing instr(TargetName, '.') > 0 guard skips a +// non-dotted target for free. It is also the spelling a user has in hand: +// `show references to combobox` is what you type after `describe widget +// combobox`, and it is the keyword the page body uses. +// +// The widget ID is not lost — it goes in TargetId, which is what that column is +// for. Two packages shipping the same MDL name would share a TargetName and be +// told apart by TargetId; SHOW REFERENCES would list both, which is a better +// failure than being unable to name the widget at all. +// +// # One edge per page, not per instance +// +// DISTINCT collapses the seven comboboxes on a page into one edge, matching the +// four sibling projections in buildReferences. Per-instance rows would say +// nothing SHOW REFERENCES or SHOW IMPACT could use, and CATALOG.WIDGETS already +// holds the instances. +func insertWidgetRefs(tx CatalogTx, projectID, snapshotID string) (int, error) { + res, err := tx.Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, TargetName, RefKind, ModuleName, ProjectId, SnapshotId) + SELECT DISTINCT w.ContainerType, '', w.ContainerQualifiedName, + 'WIDGET', d.WidgetId, d.MdlName, ?, w.ModuleName, ?, ? + FROM widgets_data w + JOIN widget_definitions_data d ON d.WidgetId = w.WidgetType + WHERE w.ContainerQualifiedName != '' AND d.MdlName != ''`, + RefKindWidget, projectID, snapshotID) + if err != nil { + return 0, err + } + n, err := res.RowsAffected() + if err != nil { + return 0, err + } + return int(n), nil +} diff --git a/mdl/catalog/builder_widget_refs_test.go b/mdl/catalog/builder_widget_refs_test.go new file mode 100644 index 0000000000..5f8b996d2b --- /dev/null +++ b/mdl/catalog/builder_widget_refs_test.go @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "testing" +) + +// seedWidgetRefFixture builds a small project in an in-memory catalog: +// +// Sales.OrderList 3 comboboxes, 1 datagrid, 2 dynamic texts (built-in) +// Sales.OrderForm 1 combobox +// Sales.AddressSnip 1 combobox (a SNIPPET, not a page) +// +// The three comboboxes on one page are what proves DISTINCT; the dynamic texts +// are what proves a built-in gets no edge. +func seedWidgetRefFixture(t *testing.T) *Catalog { + t.Helper() + cat, err := New() + if err != nil { + t.Fatalf("New: %v", err) + } + t.Cleanup(func() { cat.Close() }) + db := cat.CatalogDB() + + defs := []struct{ id, mdl string }{ + {"com.mendix.widget.web.combobox.Combobox", "COMBOBOX"}, + {"com.mendix.widget.web.datagrid.Datagrid", "DATAGRID"}, + {"com.acme.widget.Unused.Unused", "UNUSED"}, + } + for _, d := range defs { + if _, err := db.Exec( + `INSERT INTO widget_definitions_data (WidgetId, MdlName, WidgetKind, ProjectId, SnapshotId) + VALUES (?, ?, 'pluggable', 'p', 's')`, d.id, d.mdl); err != nil { + t.Fatalf("seed definition %s: %v", d.id, err) + } + } + + widgets := []struct{ id, wtype, container, ctype string }{ + {"w1", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderList", "PAGE"}, + {"w2", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderList", "PAGE"}, + {"w3", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderList", "PAGE"}, + {"w4", "com.mendix.widget.web.datagrid.Datagrid", "Sales.OrderList", "PAGE"}, + {"w5", "Forms$DynamicText", "Sales.OrderList", "PAGE"}, + {"w6", "Forms$DynamicText", "Sales.OrderList", "PAGE"}, + {"w7", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderForm", "PAGE"}, + {"w8", "com.mendix.widget.web.combobox.Combobox", "Sales.AddressSnip", "SNIPPET"}, + } + for _, w := range widgets { + if _, err := db.Exec( + `INSERT INTO widgets_data (Id, Name, WidgetType, ContainerQualifiedName, ContainerType, ModuleName, ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, 'Sales', 'p', 's')`, + w.id, w.id, w.wtype, w.container, w.ctype); err != nil { + t.Fatalf("seed widget %s: %v", w.id, err) + } + } + return cat +} + +func runInsertWidgetRefs(t *testing.T, cat *Catalog) int { + t.Helper() + tx, err := cat.CatalogDB().Begin() + if err != nil { + t.Fatalf("Begin: %v", err) + } + n, err := insertWidgetRefs(tx, "p", "s") + if err != nil { + tx.Rollback() + t.Fatalf("insertWidgetRefs: %v", err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("Commit: %v", err) + } + return n +} + +// The edge itself: one row per container x widget definition, named by the MDL +// name, carrying the widget ID, and covering snippets as well as pages. +func TestInsertWidgetRefs_EmitsOneEdgePerContainer(t *testing.T) { + cat := seedWidgetRefFixture(t) + if n := runInsertWidgetRefs(t, cat); n != 4 { + t.Fatalf("inserted %d edges, want 4: OrderList x COMBOBOX, OrderList x DATAGRID, OrderForm x COMBOBOX, AddressSnip x COMBOBOX", n) + } +} + +func TestInsertWidgetRefs_Rows(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + rows, err := cat.CatalogDB().Query( + `SELECT SourceType, SourceName, TargetType, TargetName, TargetId, RefKind + FROM refs ORDER BY SourceName, TargetName`) + if err != nil { + t.Fatalf("query refs: %v", err) + } + defer rows.Close() + + type row struct{ srcType, src, tgtType, tgt, tgtID, kind string } + var got []row + for rows.Next() { + var r row + if err := rows.Scan(&r.srcType, &r.src, &r.tgtType, &r.tgt, &r.tgtID, &r.kind); err != nil { + t.Fatalf("scan: %v", err) + } + got = append(got, r) + } + + want := []row{ + {"SNIPPET", "Sales.AddressSnip", "WIDGET", "COMBOBOX", "com.mendix.widget.web.combobox.Combobox", RefKindWidget}, + {"PAGE", "Sales.OrderForm", "WIDGET", "COMBOBOX", "com.mendix.widget.web.combobox.Combobox", RefKindWidget}, + {"PAGE", "Sales.OrderList", "WIDGET", "COMBOBOX", "com.mendix.widget.web.combobox.Combobox", RefKindWidget}, + {"PAGE", "Sales.OrderList", "WIDGET", "DATAGRID", "com.mendix.widget.web.datagrid.Datagrid", RefKindWidget}, + } + if len(got) != len(want) { + t.Fatalf("got %d rows, want %d:\n got: %v\nwant: %v", len(got), len(want), got, want) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("row %d:\n got %+v\nwant %+v", i, got[i], want[i]) + } + } +} + +// Three comboboxes on one page are one edge, not three. Without DISTINCT this +// test sees 6 rows. +func TestInsertWidgetRefs_CollapsesInstances(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var n int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE SourceName = 'Sales.OrderList' AND TargetName = 'COMBOBOX'`, + ).Scan(&n); err != nil { + t.Fatalf("count: %v", err) + } + if n != 1 { + t.Errorf("Sales.OrderList -> COMBOBOX = %d edges, want 1 — the page has three combobox instances", n) + } +} + +// A built-in widget stores its BSON $Type in the same column and has no +// definition. It must not produce an edge to a target nothing can resolve. The +// control is in the same fixture: the pluggable widgets on that same page DO +// get edges, so "no built-in edge" cannot pass by emitting nothing at all. +func TestInsertWidgetRefs_SkipsBuiltins(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var builtin, pluggable int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE TargetName LIKE 'Forms$%' OR TargetName = 'DynamicText'`, + ).Scan(&builtin); err != nil { + t.Fatalf("count builtin: %v", err) + } + if builtin != 0 { + t.Errorf("built-in widgets produced %d edges, want 0", builtin) + } + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE SourceName = 'Sales.OrderList'`, + ).Scan(&pluggable); err != nil { + t.Fatalf("count pluggable: %v", err) + } + if pluggable != 2 { + t.Errorf("control: Sales.OrderList has %d edges, want 2 — if this is 0 the test above proves nothing", pluggable) + } +} + +// An installed .mpk no page uses gets no edge, which is what makes "unused +// widget package" answerable. +func TestInsertWidgetRefs_UnusedDefinitionHasNoEdge(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var n int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE TargetName = 'UNUSED'`).Scan(&n); err != nil { + t.Fatalf("count: %v", err) + } + if n != 0 { + t.Errorf("UNUSED has %d inbound edges, want 0", n) + } +} + +// Why TargetName is the MDL name and not the widget ID. +// +// graph_module_coupling and graph_module_cohesion derive a module by taking +// everything before the FIRST dot. That is sound for a qualified name and +// nonsense for a dotted widget ID: measured on testdata/expr-checker, using the +// widget ID invented a module called "com" carrying 14 edges from three real +// modules. A non-dotted MDL name is skipped by those views' own +// instr(TargetName, '.') > 0 guard. +// +// The control is the second half: with the widget ID written into the same +// fixture, the fake module DOES appear — so this asserts a property of the +// choice, not of the fixture. +func TestInsertWidgetRefs_MdlNameKeepsModuleViewsClean(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + countCoupling := func(target string) int { + var n int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM graph_module_coupling WHERE TargetModule = ?`, target, + ).Scan(&n); err != nil { + t.Fatalf("query graph_module_coupling: %v", err) + } + return n + } + + if n := countCoupling("com"); n != 0 { + t.Errorf("graph_module_coupling has %d rows for a module 'com', want 0", n) + } + + // Control: the widget ID as TargetName does invent that module. + if _, err := cat.CatalogDB().Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, TargetName, RefKind, ModuleName, ProjectId, SnapshotId) + VALUES ('PAGE', '', 'Sales.OrderList', 'WIDGET', '', 'com.mendix.widget.web.combobox.Combobox', ?, 'Sales', 'p', 's')`, + RefKindWidget); err != nil { + t.Fatalf("seed control row: %v", err) + } + if n := countCoupling("com"); n == 0 { + t.Error("control: a dotted widget ID as TargetName should invent a module 'com' — if it does not, this test cannot detect the problem it exists for") + } +} + +// A widget definition belongs to no Mendix module, so it must not be listed as +// an asset in graph_god_nodes, where every other row is a module-qualified +// document. The page's out-degree still counts it. +func TestWidgetRefsStayOffTheGodNodeAssetList(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var asAsset int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM graph_god_nodes WHERE Asset IN ('COMBOBOX', 'DATAGRID')`, + ).Scan(&asAsset); err != nil { + t.Fatalf("query graph_god_nodes: %v", err) + } + if asAsset != 0 { + t.Errorf("graph_god_nodes lists %d widget definitions as assets, want 0", asAsset) + } + + var outDeg int + if err := cat.CatalogDB().QueryRow( + `SELECT OutDegree FROM graph_god_nodes WHERE Asset = 'Sales.OrderList'`, + ).Scan(&outDeg); err != nil { + t.Fatalf("query OutDegree: %v", err) + } + if outDeg != 2 { + t.Errorf("Sales.OrderList OutDegree = %d, want 2 — the page's dependency on the widgets it uses is real and must survive", outDeg) + } +} diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index d4ef34b350..63adc5250c 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -7,6 +7,12 @@ package catalog // // History: // +// 11 — the `widget` edge in refs (page/snippet -> widget definition) and the +// graph_god_nodes change that keeps widget targets off the asset side. +// Both need the bump for the same reason: refs are only written by +// REFRESH CATALOG FULL and a view is CREATE VIEW IF NOT EXISTS, so a +// cached catalog would answer `show references to combobox` with +// "(no references found)" — a wrong answer, not a missing one. // 10 — the three lookups expression type checking needs and the catalog could // not answer: attributes_data.EnumerationQualifiedName (DataType says only // "Enumeration", losing which one), enumeration_values_data (the table @@ -23,7 +29,7 @@ package catalog // SnapshotSource / SourceId / SourceBranch / SourceRevision columns // from every row (issue #576). // 1 — initial flat schema with denormalized snapshot columns on every row. -const CatalogSchemaVersion = "10" +const CatalogSchemaVersion = "11" // MetaSchemaVersion is the catalog_meta key that records the schema version // the cache was built against. @@ -1297,8 +1303,14 @@ func (c *Catalog) createTables() error { // graph_god_nodes — degree centrality (most depended-upon / highest fan-out). `CREATE VIEW IF NOT EXISTS graph_god_nodes AS WITH deg AS ( + -- WIDGET targets are excluded from the ASSET side: a widget + -- definition belongs to no Mendix module, and every other row here + -- is a module-qualified document, so it would list as an asset whose + -- ModuleName is its own name (the ELSE d.Asset fallback below) and + -- whose ObjectType is NULL. A page's OUT-degree still counts the + -- widgets it uses, which is a real dependency. SELECT TargetName AS Asset, COUNT(*) AS InDeg, 0 AS OutDeg - FROM refs WHERE TargetName != '' GROUP BY TargetName + FROM refs WHERE TargetName != '' AND TargetType != 'WIDGET' GROUP BY TargetName UNION ALL SELECT SourceName AS Asset, 0 AS InDeg, COUNT(*) AS OutDeg FROM refs WHERE SourceName != '' GROUP BY SourceName diff --git a/mdl/executor/cmd_associations.go b/mdl/executor/cmd_associations.go index 4dce1c3f05..4b568eb8ac 100644 --- a/mdl/executor/cmd_associations.go +++ b/mdl/executor/cmd_associations.go @@ -340,6 +340,20 @@ func execDropAssociation(ctx *ExecContext, s *ast.DropAssociationStmt) error { ctx.trackModifiedDomainModel(module.ID, module.Name) } + // A message definition holds the association by qualified name, and nothing + // keeps the two in step — the drop leaves the definition pointing at nothing + // and mxbuild rejects the project with CE1613. Unlike the access rules above + // there is nothing to reconcile: removing the member would change the + // published contract, which is the author's decision and not mxcli's. So the + // drop is refused, naming what to fix — the same posture `drop message + // definition collection` already takes from the other side. + if remedies := messageDefinitionsUsingAssociation(ctx, s.Name.String()); len(remedies) > 0 { + return mdlerrors.NewValidation(fmt.Sprintf( + "association %s is still exposed by a message definition — dropping it would leave "+ + "the definition bound to nothing (CE1613). Remove the member first:\n %s", + s.Name.String(), strings.Join(remedies, ";\n ")+";")) + } + for _, assoc := range dm.Associations { if assoc.Name == s.Name.Name { if err := ctx.Backend.DeleteAssociation(dm.ID, assoc.ID); err != nil { diff --git a/mdl/executor/cmd_associations_mock_test.go b/mdl/executor/cmd_associations_mock_test.go index d20af8b537..5b6546b2ad 100644 --- a/mdl/executor/cmd_associations_mock_test.go +++ b/mdl/executor/cmd_associations_mock_test.go @@ -294,3 +294,132 @@ func TestExecDropAssociation_ReconcilesAccessRules(t *testing.T) { t.Errorf("output does not report the reconcile: %q", buf.String()) } } + +// Dropping an association a message definition still exposes left the +// definition pointing at nothing: mxcli said "Dropped association" and named +// neither the collection nor the definition, and `describe` went on emitting +// the dangling member, so a describe -> exec round trip carried the break +// forward. mxbuild is what caught it, as CE1613 "The selected association +// 'DelProbe.Child_Parent' no longer exists" at the definition +// (ako/mxcli-rest FINDINGS #60). +// +// The neighbouring drop already behaves this way: `drop message definition +// collection` refuses while a mapping is bound to it. This is the same refusal +// from the other side. +func TestExecDropAssociation_RefusedWhileAMessageDefinitionUsesIt(t *testing.T) { + mod := mkModule("MyModule") + ent1 := mkEntity(mod.ID, "Order") + ent2 := mkEntity(mod.ID, "Customer") + assoc := mkAssociation(mod.ID, "Order_Customer", ent1.ID, ent2.ID) + dm := mkDomainModel(mod.ID, ent1, ent2) + dm.Associations = []*domainmodel.Association{assoc} + + // The association is reached from a NESTED member, not the root's own + // children — a guard that only looked one level down would miss it. + coll := &model.MessageDefinitionCollection{ + // The container is what qualifies the collection in the remedy. An + // unqualified `alter message definition MD_Orders...` would not run. + ContainerID: mod.ID, + Name: "MD_Orders", + Definitions: []*model.MessageDefinition{{ + Name: "OrderMsg", + Root: &model.MessageDefinitionElement{ + Kind: "Entity", Entity: "MyModule.Order", + OriginalName: "Order", ExposedName: "Order", + Children: []*model.MessageDefinitionElement{{ + Kind: "Entity", Entity: "MyModule.Order", + Association: "MyModule.Self", + // The exposed name differs from the original, which is the + // trap the spelled-out remedy exists to avoid. + OriginalName: "Order", ExposedName: "Self", + Children: []*model.MessageDefinitionElement{{ + Kind: "Entity", Entity: "MyModule.Customer", + Association: "MyModule.Order_Customer", + OriginalName: "Customer", ExposedName: "Buyer", + }}, + }}, + }, + }}, + } + + deleted := false + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + DeleteAssociationFunc: func(model.ID, model.ID) error { deleted = true; return nil }, + ListMessageDefinitionCollectionsFunc: func() ([]*model.MessageDefinitionCollection, error) { + return []*model.MessageDefinitionCollection{coll}, nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + err := execDropAssociation(ctx, &ast.DropAssociationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Order_Customer"}, + }) + if err == nil { + t.Fatal("the drop was accepted while a message definition still exposed the association") + } + if deleted { + t.Error("the association was deleted despite the refusal") + } + // The refusal has to spell the remedy, not just say the association is in + // use. The member's ORIGINAL name (Customer) and the exposed path to its + // holder (Self) are different names, and an author told only "remove the + // member" reaches for the one they wrote — so mxcli writes the statement. + for _, want := range []string{ + "CE1613", + "alter message definition MyModule.MD_Orders.OrderMsg drop member Customer in Self;", + } { + if !strings.Contains(err.Error(), want) { + t.Errorf("the refusal does not mention %q: %v", want, err) + } + } +} + +// CONTROL: a collection that does NOT name the association must not block the +// drop. Without this, a guard that refused whenever any collection existed +// would pass the test above. +func TestExecDropAssociation_UnrelatedMessageDefinitionDoesNotBlock(t *testing.T) { + mod := mkModule("MyModule") + ent1 := mkEntity(mod.ID, "Order") + ent2 := mkEntity(mod.ID, "Customer") + assoc := mkAssociation(mod.ID, "Order_Customer", ent1.ID, ent2.ID) + dm := mkDomainModel(mod.ID, ent1, ent2) + dm.Associations = []*domainmodel.Association{assoc} + + coll := &model.MessageDefinitionCollection{ + Name: "MD_Other", + Definitions: []*model.MessageDefinition{{ + Name: "OtherMsg", + Root: &model.MessageDefinitionElement{ + Kind: "Entity", Entity: "MyModule.Order", + Children: []*model.MessageDefinitionElement{ + {Kind: "Attribute", Attribute: "MyModule.Order.OrderNo"}, + {Kind: "Entity", Entity: "MyModule.Customer", Association: "MyModule.Order_Somebody"}, + }, + }, + }}, + } + + deleted := false + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + DeleteAssociationFunc: func(model.ID, model.ID) error { deleted = true; return nil }, + ListMessageDefinitionCollectionsFunc: func() ([]*model.MessageDefinitionCollection, error) { + return []*model.MessageDefinitionCollection{coll}, nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + if err := execDropAssociation(ctx, &ast.DropAssociationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Order_Customer"}, + }); err != nil { + t.Fatalf("an unrelated collection blocked the drop: %v", err) + } + if !deleted { + t.Error("the association was not deleted") + } +} diff --git a/mdl/executor/cmd_enumerations.go b/mdl/executor/cmd_enumerations.go index b87d79e659..683b5968c3 100644 --- a/mdl/executor/cmd_enumerations.go +++ b/mdl/executor/cmd_enumerations.go @@ -154,8 +154,16 @@ func execAlterEnumeration(ctx *ExecContext, s *ast.AlterEnumerationStmt) error { case ast.AlterEnumAdd: for _, v := range enum.Values { if v.Name == s.ValueName { + // IF NOT EXISTS makes a script that adds a value re-runnable. + // Without it the second run errors and `exec` STOPS THERE, so + // one already-present value silently truncates the rest of the + // script. (ako/mxcli-rest FINDINGS #60) + if s.IfNotExists { + fmt.Fprintf(ctx.Output, "Value '%s' already exists on enumeration %s — skipped\n", s.ValueName, s.Name) + return nil + } return mdlerrors.NewAlreadyExistsMsg("enumeration value", s.ValueName, - fmt.Sprintf("value '%s' already exists on enumeration %s", s.ValueName, s.Name)) + fmt.Sprintf("value '%s' already exists on enumeration %s — use 'add value if not exists' to make the script re-runnable", s.ValueName, s.Name)) } } enum.Values = append(enum.Values, model.EnumerationValue{ @@ -172,6 +180,11 @@ func execAlterEnumeration(ctx *ExecContext, s *ast.AlterEnumerationStmt) error { } } if idx < 0 { + // The DROP twin of the guard above: a re-run finds the value gone. + if s.IfExists { + fmt.Fprintf(ctx.Output, "Enumeration %s has no value '%s' — skipped\n", s.Name, s.ValueName) + return nil + } return mdlerrors.NewNotFound("enumeration value", s.ValueName) } enum.Values = append(enum.Values[:idx], enum.Values[idx+1:]...) diff --git a/mdl/executor/cmd_enumerations_mock_test.go b/mdl/executor/cmd_enumerations_mock_test.go index 27278ef32d..4f8671dc31 100644 --- a/mdl/executor/cmd_enumerations_mock_test.go +++ b/mdl/executor/cmd_enumerations_mock_test.go @@ -3,6 +3,7 @@ package executor import ( + "strings" "testing" "github.com/mendixlabs/mxcli/mdl/ast" @@ -251,3 +252,109 @@ func TestAlterEnumeration_ModifyValueCaption_ValueNotFound_Mock(t *testing.T) { // Backend error: cmd_error_mock_test.go (TestShowEnumerations_Mock_BackendError) // JSON: cmd_json_mock_test.go (TestShowEnumerations_Mock_JSON) + +// `alter enumeration ... add value` had no IF NOT EXISTS, so a script that adds +// one was not re-runnable: the second run errored and `exec` STOPPED THERE, +// leaving every later statement unapplied. One already-present value silently +// truncated the rest of the script (ako/mxcli-rest FINDINGS #60). +// +// Same guard pair as ALTER ENTITY's ADD ATTRIBUTE / ADD INDEX, and for the same +// reason: a defensive drop-then-add cannot be re-run either, since the drop +// fails when the value is absent and the add when it is present. +func TestAlterEnumeration_AddValueIfNotExists_Mock(t *testing.T) { + mod := mkModule("MyModule") + enum := mkEnumeration(mod.ID, "Status", "Active", "Inactive") + h := mkHierarchy(mod) + withContainer(h, enum.ContainerID, mod.ID) + + updates := 0 + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListEnumerationsFunc: func() ([]*model.Enumeration, error) { return []*model.Enumeration{enum}, nil }, + UpdateEnumerationFunc: func(*model.Enumeration) error { updates++; return nil }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + + stmt := &ast.AlterEnumerationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Status"}, + Operation: ast.AlterEnumAdd, + ValueName: "Active", + Caption: "Active", + IfNotExists: true, + } + if err := execAlterEnumeration(ctx, stmt); err != nil { + t.Fatalf("add value if not exists errored on an existing value: %v", err) + } + if updates != 0 { + t.Error("the enumeration was rewritten for a value that was already there") + } + if !strings.Contains(buf.String(), "skipped") { + t.Errorf("the skip was silent: %q", buf.String()) + } + + // CONTROL: the bare form must still error, and must say how to make the + // script re-runnable. Without this, a guard applied unconditionally would + // pass the assertion above and silently swallow a real duplicate. + bare := *stmt + bare.IfNotExists = false + err := execAlterEnumeration(ctx, &bare) + if err == nil { + t.Fatal("the unguarded add accepted a duplicate value") + } + if !strings.Contains(err.Error(), "if not exists") { + t.Errorf("the error should name the guard: %v", err) + } + + // CONTROL: the guard must not stop a value that is genuinely new. + if err := execAlterEnumeration(ctx, &ast.AlterEnumerationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Status"}, + Operation: ast.AlterEnumAdd, + ValueName: "Archived", + IfNotExists: true, + }); err != nil { + t.Fatalf("add value if not exists rejected a new value: %v", err) + } + if updates != 1 { + t.Errorf("UpdateEnumeration called %d times, want 1 — the new value was not written", updates) + } +} + +// The DROP twin: a re-run finds the value already gone. +func TestAlterEnumeration_DropValueIfExists_Mock(t *testing.T) { + mod := mkModule("MyModule") + enum := mkEnumeration(mod.ID, "Status", "Active", "Inactive") + h := mkHierarchy(mod) + withContainer(h, enum.ContainerID, mod.ID) + + updates := 0 + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListEnumerationsFunc: func() ([]*model.Enumeration, error) { return []*model.Enumeration{enum}, nil }, + UpdateEnumerationFunc: func(*model.Enumeration) error { updates++; return nil }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + + if err := execAlterEnumeration(ctx, &ast.AlterEnumerationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Status"}, + Operation: ast.AlterEnumDrop, + ValueName: "NeverThere", + IfExists: true, + }); err != nil { + t.Fatalf("drop value if exists errored on a missing value: %v", err) + } + if updates != 0 { + t.Error("the enumeration was rewritten for a value that was not there") + } + if !strings.Contains(buf.String(), "skipped") { + t.Errorf("the skip was silent: %q", buf.String()) + } + + // CONTROL: the bare form still reports a missing value. + if err := execAlterEnumeration(ctx, &ast.AlterEnumerationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Status"}, + Operation: ast.AlterEnumDrop, + ValueName: "NeverThere", + }); err == nil { + t.Fatal("the unguarded drop accepted a value that does not exist") + } +} diff --git a/mdl/executor/cmd_messagedefinitions.go b/mdl/executor/cmd_messagedefinitions.go index c9ff1d3537..c1d0142f81 100644 --- a/mdl/executor/cmd_messagedefinitions.go +++ b/mdl/executor/cmd_messagedefinitions.go @@ -274,25 +274,34 @@ func buildMessageMember(ctx *ExecContext, m *ast.MessageMemberDef, holderQN, col } // resolveAssociationCardinality returns the MaxOccurs an exposed association -// stores, from the DIRECTION the definition traverses it in. +// stores: whether the element is a single object or a list. // -// This is the one derivation in the whole document that is not obvious, and -// getting it backwards has no build error behind it — the definition simply -// exposes a list as a single object, or the reverse. +// It is a function of the direction of traversal AND the association's type, +// and the second half was learned the hard way (ako/mxcli-rest FINDINGS #60). // -// It is NOT a function of the association's type: measured across the demo -// corpus, all 927 resolvable associations are `Reference`, yet 526 store 1 and -// 401 store -1. It tracks direction, with zero counter-examples: +// forward (holder is FROM) reverse (holder is TO) +// Reference 1 -1 +// ReferenceSet -1 -1 // -// holder is the FROM entity (child -> parent, following the FK) -> 1 (496) -// holder is the TO entity (parent -> children, in reverse) -> -1 (401) +// The reverse is always a list: many holders point at one target, so from the +// target's side the element repeats. The forward direction is where the type +// decides — a Reference gives one object per holder, a ReferenceSet gives many. // -// ako/TestApp confirms it in a single document: Mappings.Order_Customer appears -// in both of its definitions and stores 1 reaching Customer from Order and -1 -// reaching Order from Customer. +// Direction ALONE looked exceptionless because the demo corpus it was measured +// on contains no ReferenceSet: all 927 resolvable associations are `Reference`, +// yet 526 store 1 and 401 store -1, which pins the direction half and says +// nothing about the type half. ako/TestApp confirms the direction half in one +// document: Mappings.Order_Customer stores 1 reaching Customer from Order and +// -1 reaching Order from Customer. +// +// Unlike the direction half, getting the type half wrong DOES have a build +// error behind it — mxbuild reports CE6524 "The occurrence of '...' has +// changed" on the definition, plus CE0295 on any object mapping element bound +// to it. Measured on ako/mxcli-rest at 11.13.0 against a 0-error baseline. // // An association that connects the two entities in NEITHER direction is refused -// rather than defaulted. A wrong cardinality is worse than a refusal: it builds. +// rather than defaulted. A wrong cardinality is worse than a refusal: for a +// Reference it builds clean and exposes a list as a single object. func resolveAssociationCardinality(ctx *ExecContext, assocQN, holderQN, targetQN, where string) (int, error) { assoc, ok := lookupAssociation(ctx, assocQN) if !ok { @@ -302,7 +311,11 @@ func resolveAssociationCardinality(ctx *ExecContext, assocQN, holderQN, targetQN fromQN, toQN := associationEnds(ctx, assoc) switch { case fromQN == holderQN && toQN == targetQN: - // Following the foreign key: one target per holder. + // Following the reference: one target per holder for a Reference, many + // for a ReferenceSet. + if assoc.Type == domainmodel.AssociationTypeReferenceSet { + return -1, nil + } return 1, nil case toQN == holderQN && fromQN == targetQN: // The reverse: many holders point at one target, so from the target's @@ -430,6 +443,88 @@ func execDropMessageDefinitionCollection(ctx *ExecContext, s *ast.DropMessageDef return nil } +// messageDefinitionsUsingAssociation returns the definitions that expose the +// named association, each as a ready ALTER statement that would remove the +// member. +// +// A message definition is a SELECTION over the domain model, so it holds the +// association by qualified name and nothing keeps the two in step: dropping the +// association leaves the definition pointing at nothing, and mxbuild rejects the +// project with CE1613 "The selected association … no longer exists". `describe` +// goes on emitting the member, so the break survives a describe -> exec round +// trip (ako/mxcli-rest FINDINGS #60). +// +// The whole element tree is walked, not the root's own children: a definition +// nests, and the association that breaks is usually not at the top. +// +// The remedy is spelled out rather than described because the two names in it +// differ and guessing wrong is the likely outcome: `drop member` matches the +// member's ORIGINAL name (the target entity's, e.g. RateSnapshot) while an `in` +// path segment matches the EXPOSED one (e.g. Snapshots). Told only "remove the +// member", an author reaches for the name they wrote in the definition, which is +// the exposed one, and gets "message definition member not found". +func messageDefinitionsUsingAssociation(ctx *ExecContext, assocQN string) []string { + colls, err := ctx.Backend.ListMessageDefinitionCollections() + if err != nil { + return nil + } + h, herr := getHierarchy(ctx) + + var out []string + for _, c := range colls { + if c == nil { + continue + } + collName := c.Name + if herr == nil { + if m := h.GetModuleName(h.FindModuleID(c.ContainerID)); m != "" { + collName = m + "." + c.Name + } + } + for _, def := range c.Definitions { + if def == nil { + continue + } + for _, hit := range findAssociationMembers(def.Root, assocQN, nil) { + stmt := fmt.Sprintf("alter message definition %s.%s drop member %s", + collName, def.Name, hit.member) + if len(hit.path) > 0 { + stmt += " in " + strings.Join(hit.path, "/") + } + out = append(out, stmt) + } + } + } + sort.Strings(out) + return out +} + +// associationMemberHit is one exposed member that reaches through the +// association: its original name, and the exposed-name path to its holder. +type associationMemberHit struct { + member string + path []string +} + +func findAssociationMembers(n *model.MessageDefinitionElement, assocQN string, path []string) []associationMemberHit { + if n == nil { + return nil + } + var out []associationMemberHit + for _, c := range n.Children { + if c == nil { + continue + } + if c.Association == assocQN { + out = append(out, associationMemberHit{member: c.OriginalName, path: append([]string(nil), path...)}) + // No recursion into a member that is itself going away. + continue + } + out = append(out, findAssociationMembers(c, assocQN, append(path, c.ExposedName))...) + } + return out +} + // mappingsUsingCollection returns the mappings whose source is a definition in // this collection. A mapping's reference is three parts // (Module.Collection.Definition), so the collection is its prefix. diff --git a/mdl/executor/cmd_messagedefinitions_test.go b/mdl/executor/cmd_messagedefinitions_test.go index 82d16bd873..4db80d2d4d 100644 --- a/mdl/executor/cmd_messagedefinitions_test.go +++ b/mdl/executor/cmd_messagedefinitions_test.go @@ -43,13 +43,24 @@ func mdFixture(t *testing.T) (*ExecContext, *model.MessageDefinitionCollection) customer.ID = nextID("cust") customer.Attributes = []*domainmodel.Attribute{mkAttr("Name", &domainmodel.StringAttributeType{})} + tag := &domainmodel.Entity{Name: "Tag", Persistable: true} + tag.ID = nextID("tag") + tag.Attributes = []*domainmodel.Attribute{mkAttr("Label", &domainmodel.StringAttributeType{})} + // ParentID is the FROM entity (the FK owner); ChildID the TO entity. - assoc := &domainmodel.Association{Name: "Order_Customer", ParentID: order.ID, ChildID: customer.ID} + assoc := &domainmodel.Association{Name: "Order_Customer", ParentID: order.ID, ChildID: customer.ID, + Type: domainmodel.AssociationTypeReference} assoc.ID = nextID("assoc") + // The same shape as a set: one Order has many Tags, so BOTH directions are + // lists. This is the association the corpus had no example of. + tags := &domainmodel.Association{Name: "Order_Tag", ParentID: order.ID, ChildID: tag.ID, + Type: domainmodel.AssociationTypeReferenceSet} + tags.ID = nextID("tags") + dm := &domainmodel.DomainModel{ContainerID: mod.ID, - Entities: []*domainmodel.Entity{base, order, customer}, - Associations: []*domainmodel.Association{assoc}, + Entities: []*domainmodel.Entity{base, order, customer, tag}, + Associations: []*domainmodel.Association{assoc, tags}, } dm.ID = nextID("dm") h := mkHierarchy(mod) @@ -139,6 +150,55 @@ func TestAssociationCardinalityFollowsTheDirection(t *testing.T) { } } +// TestReferenceSetIsAListInBothDirections is the half the direction rule got +// wrong (ako/mxcli-rest FINDINGS #60). +// +// Direction alone is the right rule for a Reference and the wrong one for a +// ReferenceSet: a set is many in BOTH directions, so the forward traversal is a +// list too. The demo corpus the direction rule was measured on contains no +// ReferenceSet at all (927 of 927 are Reference), which is why it read as +// exceptionless. +// +// Unlike the Reference case, this one HAS a build error behind it: mxbuild +// reports CE6524 "The occurrence of '...' has changed" on the definition, and +// CE0295 on any object mapping element bound to it. Measured on +// ako/mxcli-rest's RestLab.RateSnapshot_ExchangeRate at 11.13.0, against a +// baseline of 0 errors. +func TestReferenceSetIsAListInBothDirections(t *testing.T) { + // Forward: Order is the FROM entity, and reaching its Tags gives a list. + c, err := runCreate(t, mdCreate(&ast.MessageDefinitionDef{ + Name: "OrderMsg", + Entity: ast.QualifiedName{Module: "Sales", Name: "Order"}, + Members: []*ast.MessageMemberDef{assocMember("Order_Tag", "Tag", attrMember("Label"))}, + })) + if err != nil { + t.Fatalf("create: %v", err) + } + child := c.Definitions[0].Root.Children[0] + if child.MaxOccurs != -1 { + t.Errorf("Order -> Tag MaxOccurs = %d, want -1: a ReferenceSet is a list in the forward direction too", child.MaxOccurs) + } + // ExposedItemName is set exactly when the element repeats — 461 of 461 — so + // the cardinality being wrong took the item name with it. + if child.ExposedItemName != "Tag" { + t.Errorf("ExposedItemName = %q, want Tag", child.ExposedItemName) + } + + // CONTROL: the reverse was already right, and must stay right. If this were + // the only assertion, a fix that returned -1 unconditionally would pass. + c, err = runCreate(t, mdCreate(&ast.MessageDefinitionDef{ + Name: "TagMsg", + Entity: ast.QualifiedName{Module: "Sales", Name: "Tag"}, + Members: []*ast.MessageMemberDef{assocMember("Order_Tag", "Order", attrMember("OrderId"))}, + })) + if err != nil { + t.Fatalf("create: %v", err) + } + if got := c.Definitions[0].Root.Children[0].MaxOccurs; got != -1 { + t.Errorf("Tag -> Order MaxOccurs = %d, want -1 (the reverse of a set)", got) + } +} + // TestAssociationThatConnectsNeitherWayIsRefused pins the refusal. Defaulting // to 1 would build, and be wrong. func TestAssociationThatConnectsNeitherWayIsRefused(t *testing.T) { diff --git a/mdl/executor/cmd_navigation.go b/mdl/executor/cmd_navigation.go index 5c6a36c201..543c0ee58b 100644 --- a/mdl/executor/cmd_navigation.go +++ b/mdl/executor/cmd_navigation.go @@ -117,8 +117,10 @@ func execAlterNavigation(ctx *ExecContext, s *ast.AlterNavigationStmt) error { // convertMenuItemDef converts an AST NavMenuItemDef to a writer NavMenuItemSpec. func convertMenuItemDef(def ast.NavMenuItemDef) types.NavMenuItemSpec { spec := types.NavMenuItemSpec{ - Caption: def.Caption, - Icon: def.Icon, + Caption: def.Caption, + Icon: def.Icon, + IconKind: def.IconKind, + IconCode: def.IconCode, } if def.Page != nil { spec.Page = def.Page.String() @@ -419,20 +421,49 @@ func printMenuMDL(w io.Writer, items []*types.NavMenuItem, depth int, reproducer // menuItemIconMDL renders the ICON clause for a menu item, or "" when there is // nothing CREATE NAVIGATION can reproduce. +// +// All three of Mendix's icon elements have a form now. Only the collection one +// used to, so DESCRIBE emitted a comment for a glyph or image icon — and since +// CREATE NAVIGATION is a full replacement, re-running that output DELETED the +// icon it had just declined to describe. func menuItemIconMDL(item *types.NavMenuItem) string { - if item.Icon == "" || !strings.HasSuffix(item.IconType, "IconCollectionIcon") { - return "" + switch types.MenuIconKindOf(item.IconType) { + case types.MenuIconGlyph: + // The code is the whole identity of a glyph. Without it there is nothing + // to emit that would rebuild the same icon, so fall through to the note. + if item.IconCode == 0 { + return "" + } + return fmt.Sprintf(" icon glyph %d", item.IconCode) + case types.MenuIconImage: + if item.Icon == "" { + return "" + } + return " icon image " + quoteQualifiedName(item.Icon) + case types.MenuIconCollection: + if item.Icon == "" { + return "" + } + return " icon " + quoteQualifiedName(item.Icon) } - return " icon " + quoteQualifiedName(item.Icon) + return "" } -// menuItemIconNote flags an icon DESCRIBE cannot round-trip, so re-running the -// output loses it visibly rather than silently. CREATE NAVIGATION writes only -// Forms$IconCollectionIcon; a glyph icon (numeric Code) or an image icon -// (pointing into an image collection, not an icon collection) is a different -// element and would have to be guessed at. +// menuItemIconNote flags an icon DESCRIBE still cannot round-trip, so re-running +// the output loses it visibly rather than silently. +// +// All three icon elements are reproducible now, so this fires only on what is +// genuinely beyond the language: a stored $Type this build does not know, or a +// variant whose payload is missing (a glyph with no Code, a named icon with no +// name) — where emitting a clause would rebuild a DIFFERENT icon rather than the +// same one. Guessing between polymorphic variants is the failure mode that +// produces a document mxbuild accepts and Studio Pro cannot open. func menuItemIconNote(item *types.NavMenuItem, reproducer string) string { - if item.IconType == "" || strings.HasSuffix(item.IconType, "IconCollectionIcon") { + if item.IconType == "" { + return "" + } + // If there is a clause for it, there is nothing to flag. + if menuItemIconMDL(item) != "" { return "" } target := item.Icon diff --git a/mdl/executor/cmd_navigation_icon_roundtrip_test.go b/mdl/executor/cmd_navigation_icon_roundtrip_test.go new file mode 100644 index 0000000000..0565241007 --- /dev/null +++ b/mdl/executor/cmd_navigation_icon_roundtrip_test.go @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// DESCRIBE -> exec must not destroy a menu icon. +// +// # The failure this pins +// +// Measured on testdata/expr-checker, whose Home item carries a glyph icon: +// +// describe menu item 'Home' page …; +// -- icon a numeric glyph code (Forms$GlyphIcon) is not reproducible … +// exec Navigation profile 'Responsive' updated. +// describe menu item 'Home' page …; <- comment gone: the icon was DELETED +// +// CREATE NAVIGATION is a full replacement, so an icon the writer could not emit +// was an icon the statement removed. Exit 0, success message, silent loss — the +// same shape as the pluggable-widget body loss in mendixlabs/mxcli#1036. +// +// # Why this test is at this layer +// +// The round trip is describe -> parse -> write, and the defect lived in the fact +// that the three stages disagreed about what an icon IS. Asserting on the MDL +// text is what catches that: the emitted clause has to carry enough to rebuild +// the SAME element, not merely something that parses. +func TestMenuIconMDL_RoundTripsEveryVariant(t *testing.T) { + cases := []struct { + name string + item types.NavMenuItem + want string + }{ + { + "collection", + types.NavMenuItem{IconType: "Forms$IconCollectionIcon", Icon: "Atlas_Core.Atlas.home"}, + " icon Atlas_Core.Atlas.home", + }, + { + // The one that was being destroyed. The code IS the glyph's identity; + // reading only the $Type left DESCRIBE with nothing to say. + "glyph", + types.NavMenuItem{IconType: "Forms$GlyphIcon", IconCode: 57377}, + " icon glyph 57377", + }, + { + // An image icon points into an IMAGE collection, a different document + // from an icon collection — so it needs its own keyword, or replay + // would rebuild it as the wrong element. + "image", + types.NavMenuItem{IconType: "Forms$ImageIcon", Icon: "System.Images.Close"}, + " icon image System.Images.Close", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := menuItemIconMDL(&tc.item) + if got != tc.want { + t.Fatalf("menuItemIconMDL = %q, want %q", got, tc.want) + } + // It must also re-parse to the same kind, or DESCRIBE emits something + // that reads back as a different icon — which is what the bare form + // would have done for an image icon. + if note := menuItemIconNote(&tc.item, "CREATE NAVIGATION"); note != "" { + t.Errorf("still flagged as unreproducible: %q", note) + } + }) + } +} + +// The control. Without a case that still cannot be rebuilt, "emits a clause for +// everything" and "correctly reproduces everything" look identical — and the +// wrong one of those is how an unknown future variant would get silently +// converted into something else. +func TestMenuIconMDL_DeclinesWhatItCannotRebuild(t *testing.T) { + for _, item := range []types.NavMenuItem{ + // The code is the glyph's whole identity; without it there is nothing to + // emit that rebuilds the same icon. + {IconType: "Forms$GlyphIcon"}, + // A $Type this build does not know must never be guessed at. + {IconType: "Forms$SomeFutureIcon", Icon: "M.X.y"}, + } { + if got := menuItemIconMDL(&item); got != "" { + t.Errorf("%s: emitted %q, want no clause", item.IconType, got) + } + if note := menuItemIconNote(&item, "CREATE NAVIGATION"); !strings.Contains(note, "not reproducible") { + t.Errorf("%s: dropped silently instead of being flagged: %q", item.IconType, note) + } + } +} diff --git a/mdl/executor/cmd_navigation_icon_test.go b/mdl/executor/cmd_navigation_icon_test.go index 8f6cc5189e..c66676db33 100644 --- a/mdl/executor/cmd_navigation_icon_test.go +++ b/mdl/executor/cmd_navigation_icon_test.go @@ -70,27 +70,68 @@ func TestPrintMenuMDL_RoundTripsASubMenuIcon(t *testing.T) { } } -// The other two variants are real and appear in Studio Pro-authored projects, -// but CREATE NAVIGATION cannot write them. Emitting `icon '…'` for an ImageIcon -// would convert it to an IconCollectionIcon on replay — a silent variant swap. -// The loss has to be visible instead. -func TestPrintMenuMDL_FlagsAnIconItCannotReproduce(t *testing.T) { - for _, tc := range []struct{ name, iconType, icon, wantIn string }{ - {"image icon", "Forms$ImageIcon", "System.Images.Close", "System.Images.Close"}, - {"glyph icon", "Forms$GlyphIcon", "", "a numeric glyph code"}, +// All three variants round-trip now. They used to be flagged with a comment +// instead, and because CREATE NAVIGATION is a full replacement, re-running that +// output DELETED the icon the comment had just declined to describe — measured +// on testdata/expr-checker, a glyph icon destroyed at exit 0. +// +// Each form is emitted with its own keyword, so replay rebuilds the same +// ELEMENT. Writing `icon System.Images.Close` for an ImageIcon would have +// converted it to an IconCollectionIcon: a silent variant swap, which is why the +// bare form was not simply widened to cover all three. +func TestPrintMenuMDL_EmitsEachIconVariant(t *testing.T) { + for _, tc := range []struct{ name, iconType, icon, want string }{ + {"collection icon", "Forms$IconCollectionIcon", "Atlas_Core.Atlas.home", " icon Atlas_Core.Atlas.home"}, + {"image icon", "Forms$ImageIcon", "System.Images.Close", " icon image System.Images.Close"}, + } { + t.Run(tc.name, func(t *testing.T) { + got := menuMDL([]*types.NavMenuItem{{ + Caption: "Close", Page: "M.Close", Icon: tc.icon, IconType: tc.iconType, + }}) + if !strings.Contains(got, tc.want) { + t.Errorf("got %q, want it to contain %q", got, tc.want) + } + if strings.Contains(got, "-- icon") { + t.Errorf("still flagged as unreproducible: %q", got) + } + }) + } + + t.Run("glyph icon", func(t *testing.T) { + got := menuMDL([]*types.NavMenuItem{{ + Caption: "Close", Page: "M.Close", IconType: "Forms$GlyphIcon", IconCode: 57345, + }}) + if !strings.Contains(got, " icon glyph 57345") { + t.Errorf("got %q, want it to contain ` icon glyph 57345`", got) + } + if strings.Contains(got, "-- icon") { + t.Errorf("still flagged as unreproducible: %q", got) + } + }) +} + +// The note survives for what is genuinely beyond the language, and that is the +// control: without a case that still flags, "emits everything" and "flags +// nothing" are indistinguishable. +// +// A glyph with no Code cannot be rebuilt — the code IS the glyph's identity — and +// a $Type this build does not know must never be guessed at, because emitting a +// clause for it would rebuild a different element. +func TestPrintMenuMDL_StillFlagsWhatItCannotRebuild(t *testing.T) { + for _, tc := range []struct{ name, iconType, icon string }{ + {"glyph with no code", "Forms$GlyphIcon", ""}, + {"unknown variant", "Forms$SomeFutureIcon", "M.X.y"}, } { t.Run(tc.name, func(t *testing.T) { got := menuMDL([]*types.NavMenuItem{{ Caption: "Close", Page: "M.Close", Icon: tc.icon, IconType: tc.iconType, }}) - // Check the statement line only — the note below it also says "icon". stmt := strings.SplitN(got, "\n", 2)[0] if strings.Contains(stmt, " icon ") { - t.Errorf("emitted an ICON clause for %s, which replay would convert: %q", - tc.iconType, got) + t.Errorf("emitted a clause for %s, which replay could not rebuild: %q", tc.iconType, got) } - if !strings.Contains(got, "-- icon") || !strings.Contains(got, tc.wantIn) { - t.Errorf("the unreproducible icon was dropped silently: %q", got) + if !strings.Contains(got, "-- icon") { + t.Errorf("dropped silently: %q", got) } }) } diff --git a/mdl/executor/cmd_pages_builder_assoc_pagelevel_test.go b/mdl/executor/cmd_pages_builder_assoc_pagelevel_test.go new file mode 100644 index 0000000000..7bac698e90 --- /dev/null +++ b/mdl/executor/cmd_pages_builder_assoc_pagelevel_test.go @@ -0,0 +1,145 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// An association datasource at PAGE level typed its rows as the entity it was +// navigating AWAY from (mendixlabs/mxcli#1045): +// +// datagrid gA (datasource: $Customer/Bench.Order_Customer) { … OrderNo … } +// mx check -> [CE1613] "The selected attribute 'Bench.Customer.OrderNo' +// no longer exists." at Columns (1/1) of data grid 2 'gA' +// +// The destination is resolved as "the end opposite the context", and the context +// used was pb.entityContext — the ENCLOSING data container's entity. That is +// right inside a data view and empty at page level, where nothing encloses the +// widget, so neither end matched and the last-resort fallback returned the TO +// side. A named context variable answers the question directly. + +// assocPageBuilder is Order --Order_Customer--> Customer, with $Customer a page +// parameter. Order is the FROM (parent) end, Customer the TO (child) end, so +// traversing FROM Customer must reach Order. +func assocPageBuilder(entityContext string) *pageBuilder { + const ( + modID = model.ID("mod-bench") + orderID = model.ID("e-order") + custID = model.ID("e-cust") + ) + return &pageBuilder{ + entityContext: entityContext, + paramEntityNames: map[string]string{ + "Customer": "Bench.Customer", + }, + execCache: &executorCache{ + hierarchy: &ContainerHierarchy{moduleNames: map[model.ID]string{modID: "Bench"}}, + domainModels: []*domainmodel.DomainModel{{ + ContainerID: modID, + Entities: []*domainmodel.Entity{ + {BaseElement: model.BaseElement{ID: orderID}, Name: "Order"}, + {BaseElement: model.BaseElement{ID: custID}, Name: "Customer"}, + }, + Associations: []*domainmodel.Association{ + {Name: "Order_Customer", ParentID: orderID, ChildID: custID, + Type: domainmodel.AssociationTypeReference}, + }, + }}, + }, + } +} + +// THE REGRESSION. At page level there is no enclosing container, so the only +// thing that can say what the association is traversed from is the variable the +// author named. +func TestAssociationDataSource_PageLevelResolvesFromTheNamedVariable(t *testing.T) { + // entityContext is empty: nothing encloses a page-level widget. + ds, childCtx, err := assocPageBuilder("").buildDataSourceV3(&ast.DataSourceV3{ + Type: "association", + Reference: "Bench.Order_Customer", + ContextVariable: "Customer", + }) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if childCtx != "Bench.Order" { + t.Errorf("rows typed as %q, want Bench.Order — the grid navigates FROM "+ + "Customer, so its rows are the other end", childCtx) + } + assertEntityPath(t, ds, "Bench.Order_Customer/Bench.Order") +} + +// CONTROL: the data-view case must be unchanged. `$currentObject/Assoc` names +// no variable, so the enclosing entity is still what answers — and it did +// answer correctly before this fix, which is why the report singled out page +// level. +func TestAssociationDataSource_DataViewStillUsesTheEnclosingEntity(t *testing.T) { + ds, childCtx, err := assocPageBuilder("Bench.Customer").buildDataSourceV3(&ast.DataSourceV3{ + Type: "association", + Reference: "Bench.Order_Customer", + ContextVariable: "currentObject", + }) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if childCtx != "Bench.Order" { + t.Errorf("rows typed as %q, want Bench.Order", childCtx) + } + assertEntityPath(t, ds, "Bench.Order_Customer/Bench.Order") +} + +// CONTROL: a variable the builder does not know falls back to the enclosing +// entity, exactly as before. Guessing from an unknown name would be worse than +// the behaviour this replaces. +func TestAssociationDataSource_UnknownVariableFallsBackToTheEnclosingEntity(t *testing.T) { + _, childCtx, err := assocPageBuilder("Bench.Customer").buildDataSourceV3(&ast.DataSourceV3{ + Type: "association", + Reference: "Bench.Order_Customer", + ContextVariable: "SomethingElse", + }) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if childCtx != "Bench.Order" { + t.Errorf("rows typed as %q, want Bench.Order via the enclosing entity", childCtx) + } +} + +// CONTROL: traversing the OTHER way still resolves the other way. Without this, +// a fix that always returned the FROM end would pass the tests above. +func TestAssociationDataSource_ReverseDirectionStillResolves(t *testing.T) { + pb := assocPageBuilder("") + pb.paramEntityNames = map[string]string{"Order": "Bench.Order"} + + _, childCtx, err := pb.buildDataSourceV3(&ast.DataSourceV3{ + Type: "association", + Reference: "Bench.Order_Customer", + ContextVariable: "Order", + }) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if childCtx != "Bench.Customer" { + t.Errorf("rows typed as %q, want Bench.Customer — traversed from Order, "+ + "the destination is the other end", childCtx) + } +} + +// assertEntityPath checks the EntityPath the AssociationSource carries, which is +// what Mendix stores and therefore what mxbuild reads. +func assertEntityPath(t *testing.T, ds any, want string) { + t.Helper() + src, ok := ds.(*pages.AssociationSource) + if !ok { + t.Fatalf("datasource is %T, want *pages.AssociationSource", ds) + } + if src.EntityPath != want { + t.Errorf("EntityPath = %q, want %q", src.EntityPath, want) + } +} diff --git a/mdl/executor/cmd_pages_builder_missing_widget_test.go b/mdl/executor/cmd_pages_builder_missing_widget_test.go new file mode 100644 index 0000000000..bbb2b0d21b --- /dev/null +++ b/mdl/executor/cmd_pages_builder_missing_widget_test.go @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A widget whose package is NOT in widgets/ must not be told to run +// `widget init`. That command scans widgets/, so for a widget Studio Pro +// bundles rather than installs it can never help — which is what cost the +// reporter of mendixlabs/mxcli#1036 a debugging session. +func TestMissingWidgetMessage_UninstalledWidgetDoesNotRecommendWidgetInit(t *testing.T) { + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "widgets"), 0o755); err != nil { + t.Fatal(err) + } + got := missingWidgetMessage(dir, "com.mendix.widget.web.fileuploader.FileUploader") + + if strings.Contains(got, "run 'mxcli widget init") { + t.Errorf("recommends a command that cannot work:\n%s", got) + } + if !strings.Contains(got, "cannot help") { + t.Errorf("does not say why widget init is not the remedy:\n%s", got) + } + if !strings.Contains(got, "com.mendix.widget.web.fileuploader.FileUploader") { + t.Errorf("does not name the widget:\n%s", got) + } +} + +// The control: when the package IS installed, `widget init` is exactly the +// right remedy and must still be named. Without this, the test above passes +// against a build that simply deleted the recommendation. +func TestMissingWidgetMessage_InstalledWidgetStillRecommendsWidgetInit(t *testing.T) { + // FindMPK PARSES each .mpk rather than matching on its name, so this needs + // a real package — the fixture project ships Accordion among 33 others. + // (An empty file named after the widget is silently skipped, which is what + // the first version of this control got wrong.) + dir := filepath.Join("..", "..", "testdata", "expr-checker") + if _, err := os.Stat(filepath.Join(dir, "widgets")); err != nil { + t.Skipf("fixture widgets/ not available: %v", err) + } + + got := missingWidgetMessage(dir, "com.mendix.widget.web.accordion.Accordion") + + if !strings.Contains(got, "mxcli widget init") { + t.Errorf("installed package should still point at widget init:\n%s", got) + } + if !strings.Contains(got, "not extracted yet") { + t.Errorf("does not say the package is present but unextracted:\n%s", got) + } +} diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index d448c0587c..71dcee077d 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -5,6 +5,7 @@ package executor import ( "fmt" "log" + "path/filepath" "regexp" "strings" @@ -16,6 +17,7 @@ import ( "github.com/mendixlabs/mxcli/sdk/domainmodel" "github.com/mendixlabs/mxcli/sdk/microflows" "github.com/mendixlabs/mxcli/sdk/pages" + "github.com/mendixlabs/mxcli/sdk/widgets/mpk" ) // ============================================================================ @@ -432,7 +434,7 @@ func (pb *pageBuilder) buildWidgetV3(w *ast.WidgetV3) (pages.Widget, error) { if def, ok := pb.widgetRegistry.GetByWidgetID(widgetType); ok { return pb.buildPluggable(def, w) } - return nil, mdlerrors.NewNotFoundMsg("widget", widgetType, "no definition for widget "+widgetType+" (run 'mxcli widget init -p app.mpr')") + return nil, mdlerrors.NewNotFoundMsg("widget", widgetType, pb.missingWidgetMessage(widgetType)) } } } @@ -814,6 +816,38 @@ func (pb *pageBuilder) buildDataSourceV3(ds *ast.DataSourceV3) (pages.DataSource ctxVar = "" // implicit context — no SourceVariable in BSON } + // Which entity the association is traversed FROM. + // + // pb.entityContext is the ENCLOSING data container's entity, which is the + // right answer for `$currentObject/Assoc` inside a data view and the + // wrong one at page level: there is no enclosing container, so it is + // empty, resolveAssociationDestination matched neither end, and its + // last-resort fallback returned the association's TO side — the entity + // the grid was navigating AWAY from. The rows were then typed as the + // context entity and every column bound against it: + // + // datagrid gA (datasource: $Customer/Bench.Order_Customer) { … OrderNo … } + // mx check -> [CE1613] "The selected attribute 'Bench.Customer.OrderNo' + // no longer exists." at Columns (1/1) of data grid 2 'gA' + // + // The same path inside a data view was correct, which is what made the + // report's diagnosis land on page level specifically + // (mendixlabs/mxcli#1045). + // + // A NAMED context variable answers the question directly: `$Customer/…` + // traverses from whatever $Customer holds, whether or not anything + // encloses the widget. Falling back to pb.entityContext keeps the + // data-view case exactly as it was. + fromEntity := pb.entityContext + if ds.ContextVariable != "" && ds.ContextVariable != "currentObject" { + name := strings.TrimPrefix(ds.ContextVariable, "$") + if qn := pb.paramEntityNames[name]; qn != "" { + fromEntity = qn + } else if qn := pb.paramEntityNames["$"+name]; qn != "" { + fromEntity = qn + } + } + path := ds.Reference destEntity := "" if idx := strings.Index(path, "/"); idx >= 0 { @@ -829,10 +863,10 @@ func (pb *pageBuilder) buildDataSourceV3(ds *ast.DataSourceV3) (pages.DataSource // `EntityRefStep.set_AssociationId`: same unopenable project as an empty // DestinationEntity, a different property. Qualify with the context // entity's module, exactly as attribute-path hops do (upstream #854). - path = pb.resolveAssociationPathIn(path, pb.entityContext) + path = pb.resolveAssociationPathIn(path, fromEntity) if destEntity == "" { - destEntity = pb.resolveAssociationDestination(path, pb.entityContext) + destEntity = pb.resolveAssociationDestination(path, fromEntity) } else if _, _, ok := pb.associationEndpoints(path); !ok { // An author-supplied destination satisfies the guard below, so it is // the one path where a misspelled — or wrongly-moduled — association @@ -843,7 +877,7 @@ func (pb *pageBuilder) buildDataSourceV3(ds *ast.DataSourceV3) (pages.DataSource "writing it would produce a project Mendix cannot open; "+ "a bare name is qualified with the module of the context entity (%s), "+ "so an association declared elsewhere must be named in full", - path, ds.Reference, pb.entityContext) + path, ds.Reference, fromEntity) } // An empty DestinationEntity is a by-name reference Mendix resolves to @@ -2544,3 +2578,49 @@ func (pb *pageBuilder) buildMenuBarV3(w *ast.WidgetV3) (pages.Widget, error) { NavigationProfile: w.GetStringProp("Profile"), }, nil } + +// missingWidgetMessage explains why a widget has no definition, and — the part +// that matters — names a remedy that can actually work. +// +// The old message always said "run 'mxcli widget init -p app.mpr'". When the +// widget's package is not in the project at all — File Uploader, Events, Google +// Tag and Markdown viewer are in no blank project, measured on 11.13 — that is +// worse than unhelpful: `widget init` scans `widgets/`, the .mpk is not there, +// and re-running it can never help. Reported as the postscript to +// mendixlabs/mxcli#1036, where it cost the reporter a debugging session. +// +// The remedy is to install the widget, which is also the only way to use it in +// Studio Pro. Measured: a blank 11.13 project ships 33 widgets and none of those +// four; installing File Uploader takes widgets/ from 33 to 34, and mxcli then +// builds the page with no further action, because initPluggableEngine refreshes +// definitions from installed packages on its own. mxcli therefore ships no +// definitions for them — there is nothing to ship that the project does not +// already carry once the widget is usable at all. +// +// The distinguishing question is exactly the one FindMPK answers, and it is the +// same lookup the template loader makes before giving up. +func (pb *pageBuilder) missingWidgetMessage(widgetID string) string { + projectDir := "" + if pb.backend != nil { + projectDir = filepath.Dir(pb.backend.Path()) + } + return missingWidgetMessage(projectDir, widgetID) +} + +// missingWidgetMessage is the pure form, so the branch can be tested without +// standing up a whole backend. +func missingWidgetMessage(projectDir, widgetID string) string { + if projectDir != "" { + if found, err := mpk.FindMPK(projectDir, widgetID); err == nil && found != "" { + // The package is installed; the definition just has not been + // extracted from it yet. This is the case `widget init` exists for. + return "no definition for widget " + widgetID + + " — its package is installed but not extracted yet (run 'mxcli widget init -p app.mpr')" + } + } + return "no definition for widget " + widgetID + + " — the project has no widget package for it in widgets/." + + " 'mxcli widget init' cannot help: it scans widgets/, and the package is not there." + + " Install the widget or its module from the Marketplace; that puts the .mpk in widgets/," + + " after which mxcli picks it up automatically." +} diff --git a/mdl/executor/cmd_pages_describe.go b/mdl/executor/cmd_pages_describe.go index 21e00be477..3a7a7d0439 100644 --- a/mdl/executor/cmd_pages_describe.go +++ b/mdl/executor/cmd_pages_describe.go @@ -670,6 +670,16 @@ type rawWidget struct { // Object-list child blocks (for generic PLUGGABLEWIDGET output): chart series, // lines, scale colors, etc. Reconstructed from the widget's WidgetObject lists. ObjectLists []rawObjectList + + // ChildSlots are the widget's reconstructed child slots — fixed properties + // holding widgets, as opposed to ObjectLists' repeated items. + ChildSlots []rawChildSlot + + // OmittedContainers names container-shaped properties present in the stored + // document that DESCRIBE could not reproduce. Emitted as a comment so a + // describe -> exec round trip cannot silently delete a widget's body. + // See unreconstructedContainers. + OmittedContainers []string // Data container context: entity qualified name provided by this container EntityContext string // Full widget ID (e.g. "com.mendix.widget.custom.switch.Switch") diff --git a/mdl/executor/cmd_pages_describe_childslots.go b/mdl/executor/cmd_pages_describe_childslots.go new file mode 100644 index 0000000000..9ae003189c --- /dev/null +++ b/mdl/executor/cmd_pages_describe_childslots.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" +) + +// rawChildSlot is one child-slot property of a pluggable widget: a fixed +// property that holds widgets, as opposed to an object list that holds repeated +// items. +type rawChildSlot struct { + // Keyword is the MDL container keyword, derived from the property key the + // same way an object list's is (tagContentContainer -> tagcontentcontainer). + Keyword string + // PropertyKey is the stored key, kept so the emitted MDL can be matched back + // to the document when debugging. + PropertyKey string + Widgets []rawWidget +} + +// extractChildSlots reconstructs every child slot of a pluggable widget. +// +// # Why +// +// DESCRIBE reconstructed a Gallery's `content` and `filtersPlaceholder` by +// asking for those property keys BY NAME (extractGalleryWidgetsByPropertyKey), +// and nothing at all for any other widget. So a child slot on an arbitrary +// pluggable widget was dropped from the describe output — silently, at exit 0. +// +// That was invisible while slices 2-3 were unwritten, because MDL could not +// express such a slot in the first place. Once it could, describe -> edit -> +// exec started DELETING a widget's body: measured on a page mxcli authored +// itself, `tagcontentcontainer body { dynamictext t }` was stored correctly and +// came back as a bare head. +// +// This generalises the Gallery reader in the way the rest of this work +// generalises everything else: a child slot is any property whose Value holds a +// `Widgets` array, read off the document rather than looked up in a table of +// known widgets. A widget nobody has thought about round-trips for free. +// +// # Empty slots are skipped +// +// getBsonArrayElements strips the leading typed-array marker, so an EMPTY slot +// is length 0 here and length 1 in the raw BSON. Emitting empty slots would put +// a `slot { }` block on nearly every pluggable widget — correct, and unreadable. +func extractChildSlots(ctx *ExecContext, w map[string]any, entityContext string) []rawChildSlot { + obj, ok := w["Object"].(map[string]any) + if !ok { + return nil + } + keyMap := buildPropertyTypeKeyMap(w, true) + if len(keyMap) == 0 { + return nil + } + + var out []rawChildSlot + for _, prop := range getBsonArrayElements(obj["Properties"]) { + propMap, ok := prop.(map[string]any) + if !ok { + continue + } + value, ok := propMap["Value"].(map[string]any) + if !ok { + continue + } + widgetsArr := getBsonArrayElements(value["Widgets"]) + if len(widgetsArr) == 0 { + continue + } + key := keyMap[extractBinaryID(propMap["TypePointer"])] + if key == "" { + continue + } + + var widgets []rawWidget + for _, wgt := range widgetsArr { + wgtMap, ok := wgt.(map[string]any) + if !ok { + continue + } + widgets = append(widgets, parseRawWidget(ctx, wgtMap, entityContext)...) + } + if len(widgets) == 0 { + continue + } + + kw := strings.ToLower(deriveObjectListKeyword(key)) + if kw == "" { + kw = strings.ToLower(key) + } + out = append(out, rawChildSlot{Keyword: kw, PropertyKey: key, Widgets: widgets}) + } + + // Stable order: the BSON property order is not guaranteed to be meaningful, + // and an unstable describe makes diffs unusable (the same reasoning as the + // MDL-WIDGET07 ordering fix). + sort.Slice(out, func(i, j int) bool { return out[i].Keyword < out[j].Keyword }) + return out +} + +// outputChildSlots emits each reconstructed slot as an MDL container block. +// +// The slot NAME is synthesised. A child slot is a fixed property, not a named +// element — the document stores no name for it, and MDL requires one, so +// DESCRIBE WIDGET's usage example generates `slot1`, `slot2` for the same +// reason. Names are derived from the keyword so a re-describe is stable rather +// than renumbering on every run. +func outputChildSlots(ctx *ExecContext, slots []rawChildSlot, prefix string, indent int) { + for _, s := range slots { + fmt.Fprintf(ctx.Output, "%s%s %s {\n", prefix, s.Keyword, s.Keyword+"1") + for _, child := range s.Widgets { + outputWidgetMDLV3(ctx, child, indent+1) + } + fmt.Fprintf(ctx.Output, "%s}\n", prefix) + } +} diff --git a/mdl/executor/cmd_pages_describe_containers_test.go b/mdl/executor/cmd_pages_describe_containers_test.go new file mode 100644 index 0000000000..1cdcd5f884 --- /dev/null +++ b/mdl/executor/cmd_pages_describe_containers_test.go @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" +) + +// fullWidgetValue builds a widget-value sub-document the way the encoder +// actually writes one: EVERY field present, almost all of them empty. +// +// That shape is the whole point. extractObjectListItem tests the fields in +// order and used to `continue` as soon as a KEY EXISTED, so the first branch +// (DataSource) consumed every property and the ones below it never ran. The +// item came back with no props and the caller dropped it, taking the entire +// object list out of the DESCRIBE output. +func fullWidgetValue(typePointer string, set map[string]any) map[string]any { + v := map[string]any{ + "$ID": "v-" + typePointer, + "$Type": "CustomWidgets$WidgetValue", + "Action": map[string]any{"$Type": "Forms$NoAction"}, + "AttributeRef": map[string]any{}, + "DataSource": map[string]any{}, + "EntityRef": map[string]any{}, + "Expression": "", + "PrimitiveValue": "", + "TextTemplate": map[string]any{}, + "TranslatableValue": map[string]any{}, + "Widgets": []any{int32(3)}, + "Objects": []any{int32(3)}, + } + for k, val := range set { + v[k] = val + } + return map[string]any{"TypePointer": typePointer, "Value": v} +} + +// An item whose sub-properties carry real values must produce them, even though +// every other field of each value is present-but-empty. +func TestExtractObjectListItem_EmptyFieldsDoNotSwallowTheProperty(t *testing.T) { + nested := map[string]string{ + "p1": "attributeName", + "p2": "attributeValueType", + } + item := map[string]any{ + "Properties": []any{ + int32(3), + fullWidgetValue("p1", map[string]any{"PrimitiveValue": "data-x"}), + fullWidgetValue("p2", map[string]any{"PrimitiveValue": "expression"}), + }, + } + + got := extractObjectListItem(&ExecContext{}, item, nested) + if len(got.Props) != 2 { + t.Fatalf("got %d props, want 2 — an empty DataSource/Action/AttributeRef must not "+ + "consume the property before PrimitiveValue is reached:\n%+v", len(got.Props), got.Props) + } + byKey := map[string]string{} + for _, p := range got.Props { + byKey[p.Key] = p.Value + } + if byKey["attributeName"] != "data-x" { + t.Errorf("AttributeName = %q, want %q (props: %+v)", byKey["attributeName"], "data-x", got.Props) + } + if byKey["attributeValueType"] != "expression" { + t.Errorf("AttributeValueType = %q, want %q", byKey["attributeValueType"], "expression") + } +} + +// The control for the branch ordering: a property that genuinely IS a +// datasource must still be taken by the datasource branch and must NOT fall +// through to PrimitiveValue. Without this, "stop consuming on empty" could be +// implemented by removing the branches altogether and the test above would +// still pass. +func TestExtractObjectListItem_RealDataSourceStillWins(t *testing.T) { + nested := map[string]string{"p1": "staticDataSource"} + item := map[string]any{ + "Properties": []any{ + int32(3), + fullWidgetValue("p1", map[string]any{ + "PrimitiveValue": "SHOULD NOT BE READ", + "DataSource": map[string]any{ + "$Type": "Forms$ListenTargetSource", + "Widget": "someWidget", + }, + }), + }, + } + + got := extractObjectListItem(&ExecContext{}, item, nested) + for _, p := range got.Props { + if p.Value == "SHOULD NOT BE READ" { + t.Errorf("a real datasource property fell through to PrimitiveValue: %+v", got.Props) + } + } +} + +// A child slot is any property whose Value holds widgets — read off the +// document rather than looked up by name, so a widget nobody has thought about +// round-trips too. DESCRIBE previously reconstructed only a Gallery's `content` +// and `filtersPlaceholder`, by asking for those keys BY NAME. +func TestExtractChildSlots(t *testing.T) { + w := map[string]any{ + "Type": map[string]any{ + "ObjectType": map[string]any{ + "PropertyTypes": []any{ + int32(3), + map[string]any{"$ID": "t1", "PropertyKey": "tagContentContainer"}, + map[string]any{"$ID": "t2", "PropertyKey": "emptySlot"}, + }, + }, + }, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t1", + "Value": map[string]any{"Widgets": []any{int32(3), + map[string]any{"$Type": "Forms$DynamicText", "Name": "t"}}}, + }, + // Empty: the marker only. Emitting these would put a `slot { }` + // block on nearly every pluggable widget. + map[string]any{ + "TypePointer": "t2", + "Value": map[string]any{"Widgets": []any{int32(3)}}, + }, + }, + }, + } + + got := extractChildSlots(&ExecContext{}, w, "") + if len(got) != 1 { + t.Fatalf("got %d slots, want 1 (the populated one only): %+v", len(got), got) + } + if got[0].Keyword != "tagcontentcontainer" { + t.Errorf("Keyword = %q, want %q", got[0].Keyword, "tagcontentcontainer") + } + if len(got[0].Widgets) == 0 { + t.Error("slot reconstructed with no widgets — the recursion into parseRawWidget did not run") + } +} + +func TestExtractChildSlots_NoContainers(t *testing.T) { + if got := extractChildSlots(&ExecContext{}, map[string]any{}, ""); len(got) != 0 { + t.Errorf("got %+v, want none", got) + } +} diff --git a/mdl/executor/cmd_pages_describe_objectlist.go b/mdl/executor/cmd_pages_describe_objectlist.go index 0eea7dc9f6..66940caced 100644 --- a/mdl/executor/cmd_pages_describe_objectlist.go +++ b/mdl/executor/cmd_pages_describe_objectlist.go @@ -149,10 +149,34 @@ func extractObjectListItem(ctx *ExecContext, itemObj map[string]any, nestedMap m } // Per-item datasource (e.g. chart series `staticDataSource`). - if ds, ok := value["DataSource"].(map[string]any); ok && ds != nil { + // These branches must consume the property only when they actually + // EXTRACTED something. A widget value carries every field it could + // possibly have — Action, AttributeRef, DataSource, Expression, + // TextTemplate, PrimitiveValue — most of them empty, so a branch that + // `continue`s merely because its key EXISTS swallows the property and + // the branches below it never run. + // + // The measured culprit is the ACTION branch below: `value["Action"]` is + // present on every sub-property as a Forms$NoAction, and it continued + // unconditionally. On an HTML Element authored by mxcli, that consumed + // all six sub-properties of an `attribute` item; the item ended with + // zero Props, the caller's `len(item.Props) > 0` filter dropped it, and + // the whole `attributes` list vanished from DESCRIBE — while the list + // itself resolved perfectly (probe: list="attributes" objects=1 + // nested=6). Isolated by reverting that one branch: object lists go + // 2 -> 0. + // + // DataSource and AttributeRef are the same latent shape and are guarded + // the same way. Neither is load-bearing for the measured case. + if ds, ok := value["DataSource"].(map[string]any); ok && len(ds) > 0 { if rds := parseCustomWidgetDataSource(ctx, ds); rds != nil && rds.Reference != "" { item.DataSource = rds } + // Consume it either way. A datasource that is PRESENT but could not + // be rendered must not fall through to the scalar branches below — + // they would describe it as its PrimitiveValue, which is a wrong + // answer rather than a missing one. `len(ds) > 0` is the whole + // change: an EMPTY map means the field is simply unset. continue } // Child widgets (an Accordion group's `content` slot). A Widgets-typed @@ -173,17 +197,20 @@ func extractObjectListItem(ctx *ExecContext, itemObj map[string]any, nestedMap m // which is how MDL addresses an item action slot — there is no alias // (#956). A NoAction is the unset default and is skipped, so an // untouched item describes exactly as it did before. - if action, ok := value["Action"].(map[string]any); ok && action != nil { + if action, ok := value["Action"].(map[string]any); ok && len(action) > 0 { if t := extractString(action["$Type"]); t != "Forms$NoAction" && t != "Pages$NoAction" { if mdl := renderClientActionMDL(ctx, action); mdl != "" { item.Props = append(item.Props, rawExplicitProp{ Key: objectListMDLKey(key), Value: mdl, IsRef: true}) } + // A real action, rendered or not, is not a scalar. + continue } - continue + // A NoAction is the unset default: fall through, since the property + // may carry its value in one of the fields below. } // Attribute binding (staticXAttribute, staticYAttribute, …). - if attrRef, ok := value["AttributeRef"].(map[string]any); ok && attrRef != nil { + if attrRef, ok := value["AttributeRef"].(map[string]any); ok && len(attrRef) > 0 { if a := extractString(attrRef["Attribute"]); a != "" { item.Props = append(item.Props, rawExplicitProp{Key: objectListMDLKey(key), Value: shortAttributeName(a), IsRef: true}) } @@ -223,13 +250,25 @@ func extractObjectListItem(ctx *ExecContext, itemObj map[string]any, nestedMap m return item } -// objectListMDLKey maps a widget schema sub-property key to the MDL property name -// the DESCRIBE output uses. MDL property names are case-insensitive, so the -// canonical PascalCase form (first letter upper) round-trips to the same schema -// key on re-exec (staticXAttribute→StaticXAttribute, staticName→StaticName). +// objectListMDLKey is the MDL property name DESCRIBE emits for a widget schema +// sub-property: the schema key, verbatim. +// +// It used to upper-case the first letter. That round-tripped — MDL property +// names are case-insensitive, so `StaticName` and `staticName` both resolve — +// but it made DESCRIBE PAGE the only surface using that spelling: +// +// mxcli widget describe htmlelement attributeName (from the .mpk) +// what you write in a page attributeName +// DESCRIBE PAGE, before AttributeName +// +// Three surfaces, two spellings, for no benefit. It was invisible while only +// chart series reached this code; slice 3 put it on every widget with an object +// list, which is what made it worth fixing. +// +// Emitting the key verbatim also keeps a real distinction visible that +// PascalCase erased: `DataSource:` stays capitalised because it is MDL's own +// keyword, not a widget schema key, so the two kinds of name no longer look +// alike. func objectListMDLKey(schemaKey string) string { - if schemaKey == "" { - return schemaKey - } - return strings.ToUpper(schemaKey[:1]) + schemaKey[1:] + return schemaKey } diff --git a/mdl/executor/cmd_pages_describe_objectlist_test.go b/mdl/executor/cmd_pages_describe_objectlist_test.go index 53768116a5..0a37b04cdb 100644 --- a/mdl/executor/cmd_pages_describe_objectlist_test.go +++ b/mdl/executor/cmd_pages_describe_objectlist_test.go @@ -4,13 +4,20 @@ package executor import "testing" +// The key is emitted VERBATIM. It used to be PascalCased, which round-tripped +// (MDL property names are case-insensitive) but made DESCRIBE PAGE the only +// surface spelling it that way — `describe widget` documents `staticName` and +// that is what a person writes. func TestObjectListMDLKey(t *testing.T) { cases := map[string]string{ - "staticXAttribute": "StaticXAttribute", - "staticName": "StaticName", - "dataSet": "DataSet", - "interpolation": "Interpolation", + "staticXAttribute": "staticXAttribute", + "staticName": "staticName", + "dataSet": "dataSet", + "interpolation": "interpolation", "": "", + // Already-capitalised keys are untouched too: verbatim means verbatim, + // not lower-cased. A widget is free to name a property `Foo`. + "Foo": "Foo", } for in, want := range cases { if got := objectListMDLKey(in); got != want { @@ -67,13 +74,13 @@ func TestExtractObjectListItem_ChartSeries(t *testing.T) { isRef bool }{p.Value, p.IsRef} } - if p, ok := got["DataSet"]; !ok || p.val != "static" || p.isRef { + if p, ok := got["dataSet"]; !ok || p.val != "static" || p.isRef { t.Errorf("DataSet prop = %+v, want {static,false}", p) } - if p, ok := got["StaticXAttribute"]; !ok || p.val != "Region" || !p.isRef { + if p, ok := got["staticXAttribute"]; !ok || p.val != "Region" || !p.isRef { t.Errorf("StaticXAttribute prop = %+v, want {Region,true}", p) } - if p, ok := got["StaticName"]; !ok || p.val != "Revenue" || p.isRef { + if p, ok := got["staticName"]; !ok || p.val != "Revenue" || p.isRef { t.Errorf("StaticName prop = %+v, want {Revenue,false}", p) } // The datasource sub-property must NOT also appear as a scalar prop. diff --git a/mdl/executor/cmd_pages_describe_omitted.go b/mdl/executor/cmd_pages_describe_omitted.go new file mode 100644 index 0000000000..ca4451eaf0 --- /dev/null +++ b/mdl/executor/cmd_pages_describe_omitted.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "io" + "sort" + "strings" +) + +// unreconstructedContainers names the container-shaped properties a pluggable +// widget carries in its stored BSON that DESCRIBE did not reproduce. +// +// # Why this exists +// +// Slices 2-3 (PROPOSAL_def_driven_widget_bodies.md) made a widget's object +// lists and child slots WRITABLE from MDL. DESCRIBE PAGE cannot yet read them +// back for an arbitrary pluggable widget: extractObjectLists reconstructs the +// chart-shaped lists it was built for and returns nothing for HTML Element's +// `attributes`, and nothing reconstructs a child slot at all. +// +// Measured on a page mxcli itself authored: +// +// written htmlelement frame { attribute a1 (…); tagcontentcontainer body { … } } +// stored BSON carries `attributes` with data-x AND `tagContentContainer` +// with its DynamicText — the write path is correct +// described htmlelement frame (tagName: 'div', …) <- body gone, silently +// +// So describe -> edit -> exec deleted a widget's body and said nothing. That is +// the #965 failure class (an annotation emptying the loop body it sits in), and +// it became reachable the moment the construct could be written. +// +// Reconstructing them is a separate piece of work. Until then the honest +// behaviour is the one slice 4's usage example already follows: say what was +// left out rather than pretend the output is complete. A visible gap is a +// nuisance; a silent one is data loss. +// +// # Detection is from the document, not from a list +// +// A container is a property whose Value holds an `Objects` array (object list) +// or a `Widgets` array (child slot). That is read off the stored BSON, so it +// covers widgets nobody has thought about — the same reason the rest of this +// work reads definitions rather than maintaining keyword tables. +// +// Anything already reconstructed is excluded, so a chart's series list — which +// DESCRIBE does emit — produces no note. +func unreconstructedContainers(w map[string]any, reconstructed []rawObjectList, slots []rawChildSlot) []string { + obj, ok := w["Object"].(map[string]any) + if !ok { + return nil + } + keyMap := buildPropertyTypeKeyMap(w, true) + if len(keyMap) == 0 { + return nil + } + + done := make(map[string]bool, len(reconstructed)+len(slots)) + for _, ol := range reconstructed { + if ol.Keyword != "" { + done[ol.Keyword] = true + } + } + // A slot DESCRIBE now reproduces is not a loss, so it must not be named as + // one — otherwise the note would fire on exactly the case that was fixed. + for _, cs := range slots { + if cs.Keyword != "" { + done[cs.Keyword] = true + } + } + + seen := make(map[string]bool) + var out []string + for _, prop := range getBsonArrayElements(obj["Properties"]) { + propMap, ok := prop.(map[string]any) + if !ok { + continue + } + value, ok := propMap["Value"].(map[string]any) + if !ok { + continue + } + // CHILD SLOTS ONLY, and only when they hold widgets. + // + // An object list is deliberately excluded even though it is equally + // unreconstructed, because it cannot be reported without crying wolf: a + // widget template ships DEFAULT entries in its lists, and they are + // structurally identical to a user's. Measured on the probe page — + // which wrote one `attribute` and no `event` at all — the stored + // document carries one object in each, so warning on object lists named + // `event` too. A note that fires on defaults is noise, and noise trains + // people to ignore the notes that matter. + // + // A WIDGET inside a slot has no such ambiguity: a template never puts + // one there, so its presence means someone did. That is the case where + // a describe -> exec round trip destroys real work, and it is the case + // worth interrupting for. + // + // (getBsonArrayElements strips the leading typed-array marker, so a + // length of 0 here really is empty — the raw BSON array is length 1.) + if len(getBsonArrayElements(value["Widgets"])) == 0 { + continue + } + key := keyMap[extractBinaryID(propMap["TypePointer"])] + if key == "" { + continue + } + // Report the MDL keyword the author would write, not the raw property + // key, so the note names something they can act on. + kw := strings.ToLower(deriveObjectListKeyword(key)) + if kw == "" { + kw = strings.ToLower(key) + } + if done[kw] || seen[kw] { + continue + } + seen[kw] = true + out = append(out, kw) + } + sort.Strings(out) + return out +} + +// writeOmittedContainerNote emits the gap as an MDL comment, so the output +// still parses and re-executes — it simply does not carry the body, and says +// so where the body would have been. +func writeOmittedContainerNote(out io.Writer, prefix string, omitted []string) { + if len(omitted) == 0 { + return + } + fmt.Fprintf(out, "%s-- NOT SHOWN: %s — this widget stores content DESCRIBE cannot yet\n", + prefix, strings.Join(omitted, ", ")) + fmt.Fprintf(out, "%s-- reproduce. Re-running this script would DROP it. Inspect the widget with\n", prefix) + fmt.Fprintf(out, "%s-- `mxcli widget describe ` and re-add the block by hand.\n", prefix) +} diff --git a/mdl/executor/cmd_pages_describe_output.go b/mdl/executor/cmd_pages_describe_output.go index 64d1d55d9c..613bd8f48a 100644 --- a/mdl/executor/cmd_pages_describe_output.go +++ b/mdl/executor/cmd_pages_describe_output.go @@ -661,7 +661,9 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { w.OnChange != "" || len(w.NamedActions) > 0) && w.WidgetID != "" { // Generic pluggable widget with explicit properties, object-list child // blocks (chart series/lines/scaleColors), and/or an onClick action. - header := fmt.Sprintf("pluggablewidget '%s' %s", w.WidgetID, mdlIdent(w.Name)) + // The widget's own MDL name where that round-trips, else the + // explicit id form. See pluggableWidgetHeader. + header := pluggableWidgetHeader(ctx.GetWidgetRegistry(), w.WidgetID, w.Name) props := []string{} if w.Caption != "" { props = append(props, fmt.Sprintf("Label: %s", mdlQuote(w.Caption))) @@ -689,8 +691,14 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { } props = appendNamedActionProps(props, w) props = appendAppearanceProps(props, w) - if len(w.ObjectLists) == 0 { + if len(w.ObjectLists) == 0 && len(w.ChildSlots) == 0 && len(w.OmittedContainers) == 0 { formatWidgetProps(ctx.Output, prefix, header, props, "\n") + } else if len(w.ObjectLists) == 0 { + // Child slots and/or a gap to name, but no object lists. + formatWidgetProps(ctx.Output, prefix, header, props, " {\n") + outputChildSlots(ctx, w.ChildSlots, prefix+" ", indent+1) + writeOmittedContainerNote(ctx.Output, prefix+" ", w.OmittedContainers) + fmt.Fprintf(ctx.Output, "%s}\n", prefix) } else { // Emit the widget with a body holding its object-list items. formatWidgetProps(ctx.Output, prefix, header, props, " {\n") @@ -725,6 +733,11 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { formatWidgetProps(ctx.Output, childPrefix, itemHeader, itemProps, "\n") } } + // A widget can carry both kinds of container — HTML Element has + // `attributes`/`events` AND `tagContentContainer` — so the slots + // belong in this branch too, not only the one above. + outputChildSlots(ctx, w.ChildSlots, childPrefix, indent+1) + writeOmittedContainerNote(ctx.Output, childPrefix, w.OmittedContainers) fmt.Fprintf(ctx.Output, "%s}\n", prefix) } } else { diff --git a/mdl/executor/cmd_pages_describe_parse.go b/mdl/executor/cmd_pages_describe_parse.go index 77e439ae4e..96a4911730 100644 --- a/mdl/executor/cmd_pages_describe_parse.go +++ b/mdl/executor/cmd_pages_describe_parse.go @@ -455,6 +455,8 @@ func parseRawWidget(ctx *ExecContext, w map[string]any, parentEntityContext ...s if !isKnownCustomWidgetType(widget.RenderMode) { widget.ExplicitProperties = extractExplicitProperties(ctx, w) widget.ObjectLists = extractObjectLists(ctx, w) + widget.ChildSlots = extractChildSlots(ctx, w, widget.EntityContext) + widget.OmittedContainers = unreconstructedContainers(w, widget.ObjectLists, widget.ChildSlots) // onClick action (ledger #67 — reported on CustomChart): read the client // action back with full parameter mappings so a describe round-trip // re-emits it (the finding's original widget goes through this path). diff --git a/mdl/executor/cmd_search.go b/mdl/executor/cmd_search.go index dec48316fa..de3e283dda 100644 --- a/mdl/executor/cmd_search.go +++ b/mdl/executor/cmd_search.go @@ -198,8 +198,14 @@ func execShowReferences(ctx *ExecContext, s *ast.ShowStmt) error { return err } - targetName := s.Name.String() - fmt.Fprintf(ctx.Output, "\nReferences to %s\n", targetName) + typed := s.Name.String() + fmt.Fprintf(ctx.Output, "\nReferences to %s\n", typed) + + // A widget's TargetName is stored SHOUTED (COMBOBOX) while MDL keywords are + // written in lower case, so an exact-only match answers the natural spelling + // with "(no references found)" — wrong, not missing. See resolveReferenceTarget. + targetName, loose := resolveReferenceTarget(ctx, typed) + reportResolvedTarget(ctx, typed, targetName, loose) // Find all references to this target query := ` @@ -209,7 +215,7 @@ func execShowReferences(ctx *ExecContext, s *ast.ShowStmt) error { ORDER by RefKind, SourceType, SourceName ` - result, err := ctx.Catalog.Query(strings.Replace(query, "?", "'"+targetName+"'", 1)) + result, err := ctx.Catalog.Query(strings.Replace(query, "?", "'"+escapeSQLString(targetName)+"'", 1)) if err != nil { return mdlerrors.NewBackend("query references", err) } @@ -236,8 +242,11 @@ func execShowImpact(ctx *ExecContext, s *ast.ShowStmt) error { return err } - targetName := s.Name.String() - fmt.Fprintf(ctx.Output, "\nImpact analysis for %s\n", targetName) + typed := s.Name.String() + fmt.Fprintf(ctx.Output, "\nImpact analysis for %s\n", typed) + + targetName, loose := resolveReferenceTarget(ctx, typed) + reportResolvedTarget(ctx, typed, targetName, loose) // Find all direct references to this target directQuery := ` @@ -247,7 +256,7 @@ func execShowImpact(ctx *ExecContext, s *ast.ShowStmt) error { ORDER by SourceType, SourceName ` - result, err := ctx.Catalog.Query(strings.Replace(directQuery, "?", "'"+targetName+"'", 1)) + result, err := ctx.Catalog.Query(strings.Replace(directQuery, "?", "'"+escapeSQLString(targetName)+"'", 1)) if err != nil { return mdlerrors.NewBackend("query impact", err) } diff --git a/mdl/executor/cmd_security_update_test.go b/mdl/executor/cmd_security_update_test.go new file mode 100644 index 0000000000..d585e0183a --- /dev/null +++ b/mdl/executor/cmd_security_update_test.go @@ -0,0 +1,206 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// `UPDATE SECURITY` returned on the first module it could not reconcile, and one +// module always can: System, whose domain model is SYNTHESIZED rather than +// stored. So the command died there having reconciled nothing, on every MPR v2 +// project (mendixlabs/mxcli#1047): +// +// Error: failed to reconcile security for module System: load domain model +// 00000000-…-002: …/mprcontents/00/00/…002.mxunit: no such file or directory + +// updateSecurityFixture builds four modules and records which ones actually got +// reconciled. Reconciling `Broken` fails, the way System's does in a real +// project. +func updateSecurityFixture(t *testing.T) (*ExecContext, *[]string, *bytes.Buffer) { + t.Helper() + var mods []*model.Module + dms := map[model.ID]*domainmodel.DomainModel{} + // Order matters: System and Broken sit BEFORE Zulu, so a run that stops at + // the first failure never reaches it. That ordering is the regression. + for _, name := range []string{"Alpha", "System", "Broken", "Zulu"} { + m := &model.Module{Name: name} + m.ID = nextID("mod" + name) + mods = append(mods, m) + dm := &domainmodel.DomainModel{ContainerID: m.ID} + dm.ID = nextID("dm" + name) + dms[m.ID] = dm + } + + var reconciled []string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return mods, nil }, + GetDomainModelFunc: func(id model.ID) (*domainmodel.DomainModel, error) { + if dm, ok := dms[id]; ok { + return dm, nil + } + return nil, fmt.Errorf("no domain model %s", id) + }, + ReconcileMemberAccessesFunc: func(_ model.ID, moduleName string) (int, error) { + if moduleName == "Broken" { + return 0, fmt.Errorf("load domain model: no such file or directory") + } + reconciled = append(reconciled, moduleName) + return 1, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb)) + return ctx, &reconciled, buf +} + +// THE REGRESSION. One module that cannot be read must not end the run, or the +// command is inert on every project that has such a module — which is every +// project, because of System. +func TestUpdateSecurity_OneUnreadableModuleDoesNotEndTheRun(t *testing.T) { + ctx, reconciled, buf := updateSecurityFixture(t) + + if err := execUpdateSecurity(ctx, &ast.UpdateSecurityStmt{}); err != nil { + t.Fatalf("a single unreadable module aborted the whole run: %v", err) + } + // Zulu sits after both the skipped and the failing module. + if got := strings.Join(*reconciled, ","); got != "Alpha,Zulu" { + t.Errorf("reconciled %q, want Alpha,Zulu — System is skipped, Broken fails, "+ + "and neither may stop the modules after them", got) + } + // A silent skip is how "up to date" comes to mean "not looked at". + if !strings.Contains(buf.String(), "Skipped Broken") { + t.Errorf("the skipped module was not reported: %q", buf.String()) + } +} + +// System is skipped by name rather than left to fail: reconciling it is not +// merely impossible (its domain model is not stored) but wrong — its entities +// are the platform's and its access rules are not the project's to rewrite. +func TestUpdateSecurity_SkipsSystem(t *testing.T) { + ctx, reconciled, _ := updateSecurityFixture(t) + if err := execUpdateSecurity(ctx, &ast.UpdateSecurityStmt{}); err != nil { + t.Fatalf("unexpected error: %v", err) + } + for _, name := range *reconciled { + if name == "System" { + t.Error("System was reconciled — its access rules are Mendix's, not the project's") + } + } +} + +// Naming System explicitly is a mistake worth reporting. A silent skip would +// report success having done nothing, which is the failure mode this whole fix +// is about. +func TestUpdateSecurity_NamingSystemIsRefused(t *testing.T) { + ctx, _, _ := updateSecurityFixture(t) + err := execUpdateSecurity(ctx, &ast.UpdateSecurityStmt{Module: "System"}) + if err == nil { + t.Fatal("naming System was accepted") + } + if !strings.Contains(err.Error(), "platform") { + t.Errorf("the error should say why, not just refuse: %v", err) + } +} + +// A typo used to match no module, reconcile nothing, and print "All entity +// access rules are up to date" — a success message for a run that did nothing. +func TestUpdateSecurity_UnknownModuleIsRefused(t *testing.T) { + ctx, _, buf := updateSecurityFixture(t) + err := execUpdateSecurity(ctx, &ast.UpdateSecurityStmt{Module: "Nope"}) + if err == nil { + t.Fatal("an unknown module was accepted") + } + if !strings.Contains(err.Error(), "Nope") { + t.Errorf("the error should quote the name as typed: %v", err) + } + if strings.Contains(buf.String(), "up to date") { + t.Errorf("a run that matched nothing reported success: %q", buf.String()) + } +} + +// Scoping must actually scope, and match the way Mendix resolves module names. +func TestUpdateSecurity_ScopeIsHonouredAndCaseInsensitive(t *testing.T) { + ctx, reconciled, _ := updateSecurityFixture(t) + if err := execUpdateSecurity(ctx, &ast.UpdateSecurityStmt{Module: "alpha"}); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got := strings.Join(*reconciled, ","); got != "Alpha" { + t.Errorf("reconciled %q, want just Alpha", got) + } +} + +// CONTROL: with nothing skipped and nothing changed, the run still says so. +// Without this a fix that always printed a skip line would pass the tests above. +func TestUpdateSecurity_ReportsAnUpToDateProject(t *testing.T) { + mod := &model.Module{Name: "Alpha"} + mod.ID = nextID("modAlpha") + dm := &domainmodel.DomainModel{ContainerID: mod.ID} + dm.ID = nextID("dmAlpha") + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + ReconcileMemberAccessesFunc: func(model.ID, string) (int, error) { return 0, nil }, + } + ctx, buf := newMockCtx(t, withBackend(mb)) + + if err := execUpdateSecurity(ctx, &ast.UpdateSecurityStmt{}); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if !strings.Contains(buf.String(), "up to date") { + t.Errorf("an up-to-date project was not reported as such: %q", buf.String()) + } + if strings.Contains(buf.String(), "Skipped") { + t.Errorf("nothing was skipped, but a skip was reported: %q", buf.String()) + } +} + +// `update security RestLab` — without IN — used to reach the parser's error +// recovery, which consumed the module name silently: the statement parsed as one +// statement with no error and the run went PROJECT-WIDE. A scope the author +// asked for and did not get is worse than a parse error, so IN is now optional +// before the name rather than before the whole clause (mendixlabs/mxcli#1047). +func TestUpdateSecurity_BareModuleNameScopes(t *testing.T) { + for _, src := range []string{ + "update security RestLab;", + "update security in RestLab;", + } { + t.Run(src, func(t *testing.T) { + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 1 { + t.Fatalf("got %d statements, want 1", len(prog.Statements)) + } + s, ok := prog.Statements[0].(*ast.UpdateSecurityStmt) + if !ok { + t.Fatalf("statement is %T, want *ast.UpdateSecurityStmt", prog.Statements[0]) + } + if s.Module != "RestLab" { + t.Errorf("Module = %q, want RestLab — the scope was dropped, so the "+ + "run would touch every module in the project", s.Module) + } + }) + } + + // CONTROL: no name still means the whole project. + prog, errs := visitor.Build("update security;") + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if s := prog.Statements[0].(*ast.UpdateSecurityStmt); s.Module != "" { + t.Errorf("Module = %q, want empty for an unscoped run", s.Module) + } +} diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index dcad1ccf9e..49b3cbf267 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -1663,6 +1663,31 @@ func execRevokePublishedRestServiceAccess(ctx *ExecContext, s *ast.RevokePublish } // execUpdateSecurity handles UPDATE SECURITY [IN Module]. +// +// # Why one module's failure must not end the run +// +// This used to return on the first module that could not be reconciled, and one +// module always can: SYSTEM. Its domain model is SYNTHESIZED rather than stored +// — there is no unit file behind the id the module carries — so +// ReconcileMemberAccesses cannot load it, and the command died there having +// reconciled nothing: +// +// $ mxcli -p app.mpr -c 'update security' +// Error: failed to reconcile security for module System: load domain model +// 00000000-…-002: …/mprcontents/00/00/…002.mxunit: no such file or directory +// +// That made the command inert on EVERY MPR v2 project, which is the whole of +// mendixlabs/mxcli#1047's first half: it was reported as "UPDATE SECURITY does +// not fix the CE0066 it is meant to fix", and the reason is that it never ran. +// Whether it had already written some modules before dying depended on where the +// unreconcilable one fell in the list, so the failure was also non-atomic. +// +// System is now skipped by name rather than by letting it fail, because +// reconciling it is not merely impossible but wrong: its entities are the +// platform's and its access rules are not the user's to rewrite. Any OTHER +// module that cannot be read is reported and stepped over, so a project with one +// unreadable module still gets the rest reconciled — and is told which it +// missed, since a silent skip is how "up to date" comes to mean "not looked at". func execUpdateSecurity(ctx *ExecContext, s *ast.UpdateSecurityStmt) error { if !ctx.ConnectedForWrite() { return mdlerrors.NewNotConnectedWrite() @@ -1674,8 +1699,22 @@ func execUpdateSecurity(ctx *ExecContext, s *ast.UpdateSecurityStmt) error { } totalModified := 0 + matched := false + var skipped []string for _, mod := range modules { - if s.Module != "" && mod.Name != s.Module { + if s.Module != "" && !strings.EqualFold(mod.Name, s.Module) { + continue + } + matched = true + + // The platform's own module: not stored, and not ours to rewrite. + if strings.EqualFold(mod.Name, "System") { + if s.Module != "" { + return mdlerrors.NewValidation( + "System is the platform's module — its entity access rules are Mendix's, " + + "not the project's, and its domain model is not stored in the .mpr at all. " + + "There is nothing here to reconcile") + } continue } @@ -1686,7 +1725,10 @@ func execUpdateSecurity(ctx *ExecContext, s *ast.UpdateSecurityStmt) error { count, err := ctx.Backend.ReconcileMemberAccesses(dm.ID, mod.Name) if err != nil { - return mdlerrors.NewBackend(fmt.Sprintf("reconcile security for module %s", mod.Name), err) + // One module that cannot be read is not a reason to abandon the + // others, and staying quiet about it is not an option either. + skipped = append(skipped, fmt.Sprintf("%s (%v)", mod.Name, err)) + continue } if count > 0 { fmt.Fprintf(ctx.Output, "Reconciled %d access rule(s) in module %s\n", count, mod.Name) @@ -1694,7 +1736,13 @@ func execUpdateSecurity(ctx *ExecContext, s *ast.UpdateSecurityStmt) error { } } - if totalModified == 0 { + if s.Module != "" && !matched { + return mdlerrors.NewNotFound("module", s.Module) + } + for _, sk := range skipped { + fmt.Fprintf(ctx.Output, "Skipped %s — its domain model could not be read\n", sk) + } + if totalModified == 0 && len(skipped) == 0 { fmt.Fprintf(ctx.Output, "All entity access rules are up to date\n") } diff --git a/mdl/executor/describe_widget_header.go b/mdl/executor/describe_widget_header.go new file mode 100644 index 0000000000..adcfae4779 --- /dev/null +++ b/mdl/executor/describe_widget_header.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" +) + +// pluggableWidgetHeader returns the MDL header DESCRIBE PAGE should emit for a +// pluggable widget: its own MDL name where that round-trips, and the explicit +// `pluggablewidget ''` form otherwise. +// +// Item 6 of slice 2 in PROPOSAL_def_driven_widget_bodies.md. Until slices 2-3 +// the keyword form did not parse for most widgets, so DESCRIBE had no choice. +// Now that it does, emitting +// +// htmlelement frame (tagName: 'div') +// +// instead of +// +// pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' frame (…) +// +// makes describe → edit → exec produce the form a person would have written, +// which is the point of DESCRIBE being re-executable at all. +// +// # It falls back rather than guessing +// +// The bar is not "shorter" but "rebuilds the SAME widget". Two cases fail that +// and take the id form: +// +// 1. **The id resolves to no definition.** Without a project the registry +// holds only the embedded widgets, so most real widgets are unknown here — +// and an MDL name mxcli invented would not resolve on the way back in. +// 2. **Two definitions share an MDL name.** Then the name is ambiguous: the +// builder resolves it by `registry.Get(ToUpper(name))`, which can only +// return one of them, so emitting the name would silently retarget the +// widget. The id is unambiguous by construction. +// +// Case 2 is not hypothetical in principle — an MDL name is the last segment of +// a widget id, and two vendors can ship `…​.Slider`. It costs one map to rule +// out, and the alternative failure is a describe that rewrites a page onto a +// different widget. +func pluggableWidgetHeader(registry *WidgetRegistry, widgetID, name string) string { + idForm := fmt.Sprintf("pluggablewidget '%s' %s", widgetID, mdlIdent(name)) + if registry == nil || widgetID == "" { + return idForm + } + def, ok := registry.GetByWidgetID(widgetID) + if !ok || def == nil || def.MDLName == "" { + return idForm + } + // Does emitting this name rebuild the SAME widget? Ask, rather than infer. + // + // The builder resolves a bare name with registry.Get(ToUpper(name)), and the + // registry is keyed BY MDL NAME — so when two definitions claim one name the + // map keeps only the last, while GetByWidgetID keeps both. Get and + // GetByWidgetID then disagree, and emitting the name would rebuild the page + // onto the other widget. An MDL name is the last segment of a widget id, so + // two vendors shipping `….Slider` is not hypothetical. + // + // Counting definitions cannot see this (All() iterates the by-name map, so + // the loser is already gone). Round-tripping the name through the same + // lookup the builder uses can. + back, ok := registry.Get(def.MDLName) + if !ok || back == nil || back.WidgetID != widgetID { + return idForm + } + return fmt.Sprintf("%s %s", strings.ToLower(def.MDLName), mdlIdent(name)) +} diff --git a/mdl/executor/describe_widget_header_test.go b/mdl/executor/describe_widget_header_test.go new file mode 100644 index 0000000000..c71832af46 --- /dev/null +++ b/mdl/executor/describe_widget_header_test.go @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// Item 6 of slice 2: DESCRIBE emits the widget's own MDL name now that the +// keyword form parses for every widget with a definition. +func TestPluggableWidgetHeader_UsesTheMDLName(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("no registry — the embedded definitions must load, or this test proves nothing") + } + + def, ok := registry.Get("COMBOBOX") + if !ok || def == nil || def.WidgetID == "" { + t.Fatal("COMBOBOX is not an embedded definition; pick another widget for this test") + } + + got := pluggableWidgetHeader(registry, def.WidgetID, "cmb1") + want := "combobox cmb1" + if got != want { + t.Errorf("header = %q, want %q", got, want) + } +} + +// The fallbacks. Each must produce the id form, which always round-trips. +func TestPluggableWidgetHeader_FallsBackToTheIDForm(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("no registry") + } + + cases := []struct { + name string + registry *WidgetRegistry + widgetID string + why string + }{ + {"unknown id", registry, "com.acme.NotInstalled.Thing", + "an id with no definition has no MDL name to emit; inventing one would not resolve on the way back in"}, + {"no registry", nil, "com.mendix.widget.web.combobox.Combobox", + "without definitions there is nothing to resolve against"}, + {"empty id", registry, "", + "nothing to look up"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := pluggableWidgetHeader(tc.registry, tc.widgetID, "w1") + if !strings.HasPrefix(got, "pluggablewidget '") { + t.Errorf("header = %q, want the pluggablewidget id form — %s", got, tc.why) + } + }) + } +} + +// The ambiguity guard. Two definitions sharing an MDL name make the name +// unusable: the builder resolves it with registry.Get, which can return only +// one of them, so emitting the name could silently retarget the widget onto a +// different one. An MDL name is the last segment of a widget id, so two vendors +// shipping `….Slider` is not hypothetical. +func TestPluggableWidgetHeader_AmbiguousNameFallsBack(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("no registry") + } + def, ok := registry.Get("COMBOBOX") + if !ok { + t.Fatal("COMBOBOX missing") + } + + // Control first: unique today, so the name IS emitted. + if got := pluggableWidgetHeader(registry, def.WidgetID, "w1"); !strings.HasPrefix(got, "combobox ") { + t.Fatalf("control: header = %q, want the name form before a collision is introduced", got) + } + + // Introduce a second definition claiming the same MDL name. This is what a + // colliding install looks like in the registry: byWidgetID keeps both, + // byMDLName keeps only the last, so the two lookups disagree. + other := &WidgetDefinition{WidgetID: "com.acme.other.Combobox", MDLName: "combobox"} + registry.byWidgetID[other.WidgetID] = other + registry.byMDLName["COMBOBOX"] = other + if got := pluggableWidgetHeader(registry, def.WidgetID, "w1"); !strings.HasPrefix(got, "pluggablewidget '") { + t.Errorf("header = %q, want the id form once two definitions claim the MDL name — "+ + "emitting the ambiguous name could rebuild the page onto the other widget", got) + } +} + +// unreconstructedContainers turns silent data loss into a visible gap. +// +// DESCRIBE cannot yet read a child slot back for an arbitrary pluggable widget, +// so describe -> exec deleted a widget's body and said nothing. Measured on a +// page mxcli authored itself: the stored BSON carried `tagContentContainer` +// with a DynamicText, and the describe output was a bare head. +func TestUnreconstructedContainers(t *testing.T) { + // A minimal pluggable widget document: two properties, one child slot with + // a widget in it and one empty. Arrays carry the leading typed-array marker, + // which is why an EMPTY container is length 1 and not length 0 — getting + // that wrong reports every widget as lossy. + widget := map[string]any{ + "Type": map[string]any{ + "PropertyTypes": []any{ + int32(3), + map[string]any{"$ID": "t1", "PropertyKey": "tagContentContainer"}, + map[string]any{"$ID": "t2", "PropertyKey": "emptySlot"}, + }, + }, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t1", + "Value": map[string]any{ + "Widgets": []any{int32(3), map[string]any{"$Type": "Forms$DynamicText"}}, + }, + }, + map[string]any{ + "TypePointer": "t2", + "Value": map[string]any{"Widgets": []any{int32(3)}}, + }, + }, + }, + } + + got := unreconstructedContainers(widget, nil, nil) + + // The populated slot must be reported. + var sawPopulated, sawEmpty bool + for _, g := range got { + if strings.Contains(g, "tagcontent") { + sawPopulated = true + } + if strings.Contains(g, "empty") { + sawEmpty = true + } + } + if !sawPopulated { + t.Errorf("a child slot holding a widget was not reported; got %v — "+ + "this is the case where describe -> exec destroys real work", got) + } + // The control: an EMPTY slot must not be reported, or the note fires on + // every widget and stops being read. + if sawEmpty { + t.Errorf("an empty child slot was reported as lost; got %v — an array of length 1 "+ + "is the typed-array marker alone, i.e. no content", got) + } +} + +// Nothing to report on a document with no containers at all. +func TestUnreconstructedContainers_Empty(t *testing.T) { + if got := unreconstructedContainers(map[string]any{}, nil, nil); len(got) != 0 { + t.Errorf("got %v, want none", got) + } +} diff --git a/mdl/executor/exec_context.go b/mdl/executor/exec_context.go index dda34b5bf1..b62b36906e 100644 --- a/mdl/executor/exec_context.go +++ b/mdl/executor/exec_context.go @@ -62,6 +62,12 @@ type ExecContext struct { // ThemeRegistry holds cached theme design property definitions (lazy init). ThemeRegistry *ThemeRegistry + // widgetRegistry caches the widget definitions for this session. Loaded + // once via GetWidgetRegistry, because DESCRIBE PAGE consults it per widget + // and LoadWidgetRegistry reads .def.json files off disk. + widgetRegistry *WidgetRegistry + widgetRegistryLoaded bool + // Settings holds session-scoped key-value settings (SET command). Settings map[string]any @@ -285,3 +291,28 @@ func (ctx *ExecContext) ensureSqlMgr() *sqllib.Manager { } return ctx.SqlMgr } + +// GetWidgetRegistry returns the session's widget registry, loading it on first +// use. It is cached because DESCRIBE PAGE asks per widget and the load reads +// every .def.json in the project — file I/O in a per-widget path is exactly +// what the review checklist warns against. +// +// A nil result is normal and means "no definitions available": with no project +// only the embedded widgets exist, and callers must degrade rather than treat +// it as an error. +func (ctx *ExecContext) GetWidgetRegistry() *WidgetRegistry { + if ctx == nil { + return nil + } + if ctx.widgetRegistryLoaded { + return ctx.widgetRegistry + } + ctx.widgetRegistryLoaded = true + // LoadWidgetRegistry wants the .mpr PATH, not its directory — + // LoadUserDefinitions takes filepath.Dir of it internally. Passing the + // directory looks one level too high and silently finds no definitions, + // which shows up as DESCRIBE falling back to the widget-id form for every + // widget rather than as an error. + ctx.widgetRegistry = LoadWidgetRegistry(ctx.MprPath) + return ctx.widgetRegistry +} diff --git a/mdl/executor/executor_query.go b/mdl/executor/executor_query.go index e8cbfc72ae..672285bc1d 100644 --- a/mdl/executor/executor_query.go +++ b/mdl/executor/executor_query.go @@ -169,7 +169,13 @@ func execShow(ctx *ExecContext, s *ast.ShowStmt) error { } func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { - if !ctx.Connected() && s.ObjectType != ast.DescribeFragment { + // DESCRIBE WIDGET joins DESCRIBE FRAGMENT in not needing a project: a widget + // definition is not a document in the model, and mxcli's embedded set can + // answer for a built-in widget with nothing open. With a project the answer + // is better — the installed .mpk is version-accurate and covers Marketplace + // widgets — but requiring one would make the statement useless for exactly + // the "what can I write here?" question it exists to answer. + if !ctx.Connected() && s.ObjectType != ast.DescribeFragment && s.ObjectType != ast.DescribeWidget { return mdlerrors.NewNotConnected() } @@ -243,6 +249,8 @@ func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { return describeExternalEntity(ctx, s.Name) case ast.DescribeNavigation: return describeNavigation(ctx, s.Name) + case ast.DescribeWidget: + return describeWidgetStmt(ctx, s.Name.Name) case ast.DescribeWorkflow: return describeWorkflow(ctx, s.Name) case ast.DescribeBusinessEventService: @@ -340,6 +348,8 @@ func describeObjectTypeLabel(t ast.DescribeObjectType) string { return "externalentity" case ast.DescribeNavigation: return "navigation" + case ast.DescribeWidget: + return "widget" case ast.DescribeWorkflow: return "workflow" case ast.DescribeBusinessEventService: diff --git a/mdl/executor/oql_sum_unknown_test.go b/mdl/executor/oql_sum_unknown_test.go new file mode 100644 index 0000000000..a8cdad767e --- /dev/null +++ b/mdl/executor/oql_sum_unknown_test.go @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "errors" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// unresolvableCtx is the real shape of the bug: a project the checker can read, +// in which the view's SOURCE entity does not exist — because the script creates +// it in the same run, and `check --references` skips script-created objects. +func unresolvableCtx() *ExecContext { + return &ExecContext{Backend: &mock.MockBackend{ + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { + return nil, errors.New("no domain models") + }, + }} +} + +// SUM over an argument whose type could not be resolved must be Unknown, not +// Decimal — the same rule inferTypeStatic already states in its own comment and +// the project-aware path contradicted. +// +// # The failure this caused +// +// A view entity whose source entity is created BY THE SAME SCRIPT cannot have +// its attribute types resolved (check skips references to script-created +// objects), so `sum(s.Units)` resolved to Unknown and fell through to Decimal. +// mxcli then reported +// +// attribute 'Units': declared as Integer but OQL expression 'sum(s.Units)' +// returns Decimal. Fix: change to 'Units: Decimal' +// +// Measured against mxbuild 11.6.6, that hint INVERTS the truth. Two views over +// the same `sum(s.Units)` where Units is an Integer attribute: +// +// declared Integer -> 0 errors (what mxcli flagged) +// declared Decimal -> CE6770 (what mxcli told the user to write) +// +// The control that the check is not simply inert: a view declaring a String +// column over `sum(s.Amount)` DOES fail CE6770 on the same mxbuild, so mxbuild +// really does validate these and the 0-errors result above means something. +// +// So the diagnostic did not merely cry wolf, it walked a working project into a +// broken one. An unresolvable type has to stay unresolved. +func TestInferAggregateType_SumOfUnknownIsUnknown(t *testing.T) { + ctx := unresolvableCtx() + aliasMap := map[string]string{"s": "ChartExamples.Sales"} + + got := inferAggregateType(ctx, "sum(s.Units)", &OQLColumnInfo{}, aliasMap) + if got.Kind != ast.TypeUnknown { + t.Errorf("sum() over an unresolvable argument inferred %s, want Unknown — "+ + "guessing Decimal makes the checker demand a declaration mxbuild rejects (CE6770)", + formatDataTypeForError(got)) + } +} + +// The control for the fix: SUM must still PROPAGATE a type it can resolve, or +// "return Unknown" degenerates into "never check sum() at all" and the rule +// stops detecting the real CE6770 it was written for. +func TestInferAggregateType_SumPropagatesAKnownType(t *testing.T) { + ctx := unresolvableCtx() + // A literal resolves without a project, so this exercises the propagation + // branch rather than the entity lookup. + if got := inferAggregateType(ctx, "sum(1.5)", &OQLColumnInfo{}, nil); got.Kind != ast.TypeDecimal { + t.Errorf("sum(1.5) inferred %s, want Decimal — a resolvable argument type must "+ + "still propagate", formatDataTypeForError(got)) + } + if got := inferAggregateType(ctx, "sum(2)", &OQLColumnInfo{}, nil); got.Kind != ast.TypeInteger { + t.Errorf("sum(2) inferred %s, want Integer", formatDataTypeForError(got)) + } +} + +// The second control: the neighbouring aggregates keep their own rules. COUNT is +// Integer whatever its argument, AVG is Decimal whatever its argument — a fix +// that made every aggregate Unknown would pass the first test and silently turn +// the whole rule off. +func TestInferAggregateType_NeighbouringAggregatesUnchanged(t *testing.T) { + ctx := unresolvableCtx() + aliasMap := map[string]string{"s": "ChartExamples.Sales"} + + if got := inferAggregateType(ctx, "count(s.ID)", &OQLColumnInfo{}, aliasMap); got.Kind != ast.TypeInteger { + t.Errorf("count() inferred %s, want Integer", formatDataTypeForError(got)) + } + if got := inferAggregateType(ctx, "avg(s.Units)", &OQLColumnInfo{}, aliasMap); got.Kind != ast.TypeDecimal { + t.Errorf("avg() inferred %s, want Decimal", formatDataTypeForError(got)) + } +} diff --git a/mdl/executor/oql_type_inference.go b/mdl/executor/oql_type_inference.go index e13758b8c0..b90e57bc7c 100644 --- a/mdl/executor/oql_type_inference.go +++ b/mdl/executor/oql_type_inference.go @@ -694,18 +694,32 @@ func inferAggregateType(ctx *ExecContext, expr string, col *OQLColumnInfo, alias return ast.DataType{Kind: ast.TypeInteger} } - // SUM(expression) → preserves input type (Integer→Integer, else Decimal) + // SUM(expression) → preserves the input type, and stays UNKNOWN when that + // type could not be resolved. + // + // Falling back to Decimal looks harmless and is not: the argument is + // unresolvable exactly when the source entity is created by the same script + // (check skips references to script-created objects), which is the common + // shape for a view entity. Measured against mxbuild 11.6.6 on `sum(s.Units)` + // where Units is an Integer attribute: + // + // declared Integer -> 0 errors <- what mxcli flagged + // declared Decimal -> CE6770 <- what mxcli's hint told the user to write + // + // So the guess inverted the truth and the "Fix:" would break a working + // project. inferTypeStatic's SUM branch already says this in its own comment; + // this path disagreed with it. if strings.HasPrefix(upperExpr, "SUM(") { col.IsAggregate = true col.AggregateFunc = "sum" innerArg := extractFunctionArg(expr) if innerArg != "" { innerType := inferTypeFromExpression(ctx, innerArg, &OQLColumnInfo{}, aliasMap) - if innerType.Kind == ast.TypeInteger || innerType.Kind == ast.TypeLong { + if innerType.Kind != ast.TypeUnknown { return innerType } } - return ast.DataType{Kind: ast.TypeDecimal} + return ast.DataType{Kind: ast.TypeUnknown} } // AVG(expression) → always Decimal diff --git a/mdl/executor/reference_target.go b/mdl/executor/reference_target.go new file mode 100644 index 0000000000..a458c3f8f0 --- /dev/null +++ b/mdl/executor/reference_target.go @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" +) + +// resolveReferenceTarget returns the spelling of `name` that CATALOG.REFS +// actually stores, and whether it differs from what the user typed. +// +// SHOW REFERENCES TO and SHOW IMPACT OF match TargetName exactly. Every target +// used to be a module-qualified name, which a user copies verbatim from +// `show entities`, so exact matching was right and nothing needed this. +// +// The `widget` edge (slice 5 of PROPOSAL_def_driven_widget_bodies.md) breaks +// that assumption: its TargetName is the widget's MDL name, which is stored +// SHOUTED (COMBOBOX) because that is how widget_definitions_data holds it, +// while MDL keywords are case-insensitive and every example writes them in +// lower case. So the natural +// +// show references to combobox +// +// found nothing — and reported "(no references found)", which is a WRONG +// answer rather than a missing one. That is the failure mode worth spending a +// lookup to avoid: a user cannot tell it from a widget genuinely being unused. +// +// The fallback is deliberately second, never first. An exact match is returned +// untouched, so no existing answer can change; only a query that would have +// returned nothing gets a second chance. Callers report the resolved spelling +// so the user can see which name was actually matched. +func resolveReferenceTarget(ctx *ExecContext, name string) (resolved string, matchedLoosely bool) { + if ctx == nil || ctx.Catalog == nil || name == "" { + return name, false + } + + exact, err := ctx.Catalog.Query(fmt.Sprintf( + `SELECT 1 FROM refs WHERE TargetName = '%s' LIMIT 1`, escapeSQLString(name))) + if err == nil && exact.Count > 0 { + return name, false + } + + // Nothing under that spelling. Try case-insensitively, and only accept the + // answer when it is unambiguous — two targets differing only in case are a + // question this cannot answer for the user, so leave the exact (empty) + // result rather than guessing at one of them. + loose, err := ctx.Catalog.Query(fmt.Sprintf( + `SELECT DISTINCT TargetName FROM refs WHERE lower(TargetName) = lower('%s')`, + escapeSQLString(name))) + if err != nil || loose.Count != 1 || len(loose.Rows) != 1 || len(loose.Rows[0]) == 0 { + return name, false + } + match, ok := loose.Rows[0][0].(string) + if !ok || match == "" || match == name { + return name, false + } + return match, true +} + +// reportResolvedTarget tells the user which stored spelling was matched, when +// it is not the one they typed. Silent on an exact match. +func reportResolvedTarget(ctx *ExecContext, typed, resolved string, matchedLoosely bool) { + if !matchedLoosely || ctx == nil || ctx.Output == nil { + return + } + fmt.Fprintf(ctx.Output, "(matched %s)\n", strings.TrimSpace(resolved)) +} diff --git a/mdl/executor/reference_target_test.go b/mdl/executor/reference_target_test.go new file mode 100644 index 0000000000..657e48ede9 --- /dev/null +++ b/mdl/executor/reference_target_test.go @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/catalog" +) + +// seedRefTargets builds an in-memory catalog holding one widget edge (stored +// SHOUTED, as widget_definitions_data holds MDL names) and one ordinary +// module-qualified target, so the exact path and the fallback are exercised +// against the same catalog. +func seedRefTargets(t *testing.T, targets ...string) *ExecContext { + t.Helper() + cat, err := catalog.New() + if err != nil { + t.Fatalf("catalog.New: %v", err) + } + t.Cleanup(func() { cat.Close() }) + for _, tgt := range targets { + if _, err := cat.CatalogDB().Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, TargetName, RefKind, ProjectId, SnapshotId) + VALUES ('PAGE', '', 'Sales.OrderList', 'WIDGET', '', ?, 'widget', 'p', 's')`, tgt); err != nil { + t.Fatalf("seed %q: %v", tgt, err) + } + } + return &ExecContext{Catalog: cat, Output: &bytes.Buffer{}} +} + +func TestResolveReferenceTarget(t *testing.T) { + ctx := seedRefTargets(t, "COMBOBOX", "Sales.Order") + + cases := []struct { + name string + typed string + want string + wantLoose bool + reasonWhen string + }{ + {"exact widget name", "COMBOBOX", "COMBOBOX", false, + "an exact match must be returned untouched, so no existing answer changes"}, + {"lower-case widget name", "combobox", "COMBOBOX", true, + "the spelling every MDL example uses must find the stored SHOUTED name"}, + {"mixed-case widget name", "ComboBox", "COMBOBOX", true, + "MDL keywords are case-insensitive, so any casing must resolve"}, + {"exact qualified name", "Sales.Order", "Sales.Order", false, + "an ordinary target keeps exact-match behaviour"}, + {"wrong-case qualified name", "sales.order", "Sales.Order", true, + "the fallback is not widget-specific; it can only turn an empty answer into a right one"}, + {"genuine typo", "Sales.Ordr", "Sales.Ordr", false, + "a name that matches nothing in any casing must not be rewritten to something else"}, + {"unknown widget", "gallery", "gallery", false, + "a widget with no edges must stay unresolved rather than borrow another's name"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, loose := resolveReferenceTarget(ctx, tc.typed) + if got != tc.want || loose != tc.wantLoose { + t.Errorf("resolveReferenceTarget(%q) = (%q, %v), want (%q, %v) — %s", + tc.typed, got, loose, tc.want, tc.wantLoose, tc.reasonWhen) + } + }) + } +} + +// Two targets differing only in case are a question this cannot answer, so it +// must decline rather than pick one. Without the Count != 1 guard it would +// silently return whichever the database ordered first. +func TestResolveReferenceTarget_AmbiguousCaseDeclines(t *testing.T) { + ctx := seedRefTargets(t, "Sales.Order", "sales.ORDER") + + got, loose := resolveReferenceTarget(ctx, "SALES.order") + if loose || got != "SALES.order" { + t.Errorf("resolveReferenceTarget = (%q, %v), want (%q, false) — two case-variant targets must not be guessed between", + got, loose, "SALES.order") + } + + // Control: with only one variant present, the same input DOES resolve — so + // the decline above is the ambiguity guard, not a broken lookup. + single := seedRefTargets(t, "Sales.Order") + if got, loose := resolveReferenceTarget(single, "SALES.order"); !loose || got != "Sales.Order" { + t.Errorf("control: resolveReferenceTarget = (%q, %v), want (\"Sales.Order\", true)", got, loose) + } +} + +// The resolved spelling is reported, because a user who typed `combobox` and +// got results under `COMBOBOX` should be able to see which name matched. +func TestReportResolvedTarget(t *testing.T) { + var buf bytes.Buffer + ctx := &ExecContext{Output: &buf} + + reportResolvedTarget(ctx, "COMBOBOX", "COMBOBOX", false) + if buf.Len() != 0 { + t.Errorf("exact match printed %q, want nothing", buf.String()) + } + + reportResolvedTarget(ctx, "combobox", "COMBOBOX", true) + if got := buf.String(); !strings.Contains(got, "COMBOBOX") { + t.Errorf("loose match printed %q, want it to name COMBOBOX", got) + } +} + +// A nil catalog must not panic — `show references` reaches here only after +// ensureCatalog, but the helper is small enough to be called elsewhere. +func TestResolveReferenceTarget_NoCatalog(t *testing.T) { + if got, loose := resolveReferenceTarget(&ExecContext{}, "combobox"); got != "combobox" || loose { + t.Errorf("no catalog = (%q, %v), want (\"combobox\", false)", got, loose) + } + if got, loose := resolveReferenceTarget(nil, "combobox"); got != "combobox" || loose { + t.Errorf("nil ctx = (%q, %v), want (\"combobox\", false)", got, loose) + } +} diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 85e03184df..74153e1e91 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -26,6 +26,7 @@ type scriptContext struct { nanoflows map[string]bool // Nanoflows created (Module.Nanoflow) pages map[string]bool // Pages created (Module.Page) snippets map[string]bool // Snippets created (Module.Snippet) + layouts map[string]bool // Layouts created (Module.Layout) constants map[string]bool // Constants created (Module.Constant) workflows map[string]bool // Workflows created (Module.Workflow) @@ -58,6 +59,7 @@ func newScriptContext() *scriptContext { pages: make(map[string]bool), workflows: make(map[string]bool), snippets: make(map[string]bool), + layouts: make(map[string]bool), constants: make(map[string]bool), javaActions: make(map[string][]string), @@ -120,6 +122,10 @@ func (sc *scriptContext) collectDefinitions(prog *ast.Program) { if s.Name.Module != "" { sc.snippets[s.Name.String()] = true } + case *ast.CreateLayoutStmt: + if s.Name.Module != "" { + sc.layouts[s.Name.String()] = true + } case *ast.CreateWorkflowStmt: if s.Name.Module != "" { sc.workflows[s.Name.String()] = true @@ -175,6 +181,10 @@ func (sc *scriptContext) collectSingle(stmt ast.Statement) { if s.Name.Module != "" { sc.snippets[s.Name.String()] = true } + case *ast.CreateLayoutStmt: + if s.Name.Module != "" { + sc.layouts[s.Name.String()] = true + } case *ast.CreateWorkflowStmt: if s.Name.Module != "" { sc.workflows[s.Name.String()] = true @@ -294,6 +304,18 @@ func validateProgram(ctx *ExecContext, prog *ast.Program) []error { // microflow — errors whose wording sends people to the entity import, which // cannot fix either of them (mendixlabs/mxcli#1020). errors = append(errors, validateExternalActionCalls(ctx, prog)...) + // Resolve MEMBER names inside a CREATE / CHANGE against the entity they are + // assigned to. Reference checking used to stop at the document and entity + // level, so a mistyped attribute passed check and exec and surfaced as + // CE1613 at the far end of a build (mendixlabs/mxcli#1048). + for _, msg := range validateMemberReferences(ctx, prog, sc) { + errors = append(errors, mdlerrors.NewValidation(msg)) + } + // Resolve the MEMBERS inside a widget's XPath constraint. The entity in + // `database from Mod.Entity` was resolved and the `where […]` was not, so a + // member that does not exist reached mxbuild as CE1613 + // (mendixlabs/mxcli#1049). + errors = append(errors, validateXPathMembers(ctx, prog)...) return errors } @@ -386,6 +408,13 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext return err } + // An ALTER's target document must already exist. Until this, only the + // MODULE was resolved, so a misspelled document passed --references and was + // refused by exec — check was the weaker gate, which is backwards. + if err := validateAlterTarget(ctx, stmt, sc); err != nil { + return err + } + switch s := stmt.(type) { // Statements that reference modules case *ast.CreateEntityStmt: @@ -549,7 +578,7 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext s.Name.String(), strings.Join(refErrors, "\n - ")) } // Validate page context tree (parameter/selection/attribute bindings) - if ctxErrors := validatePageContextTree(s.Parameters, s.Widgets); len(ctxErrors) > 0 { + if ctxErrors := validatePageContextTree(ctx, s.Parameters, s.Widgets); len(ctxErrors) > 0 { return mdlerrors.NewValidationf("page '%s' has context errors:\n - %s", s.Name.String(), strings.Join(ctxErrors, "\n - ")) } @@ -576,7 +605,7 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext s.Name.String(), strings.Join(argErrors, "\n - ")) } // Validate snippet context tree (parameter/selection/attribute bindings) - if ctxErrors := validatePageContextTree(s.Parameters, s.Widgets); len(ctxErrors) > 0 { + if ctxErrors := validatePageContextTree(ctx, s.Parameters, s.Widgets); len(ctxErrors) > 0 { return mdlerrors.NewValidationf("snippet '%s' has context errors:\n - %s", s.Name.String(), strings.Join(ctxErrors, "\n - ")) } diff --git a/mdl/executor/validate_alter_target.go b/mdl/executor/validate_alter_target.go new file mode 100644 index 0000000000..8bfed30b03 --- /dev/null +++ b/mdl/executor/validate_alter_target.go @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" +) + +// An ALTER names a document that must already exist, and nothing checked that it +// did. `alter page RestLab."RestLab_Home"` — no such page — passed +// `mxcli check -p --references` and was then refused by `exec` with "page not +// found" (ako/mxcli-rest FINDINGS #60). +// +// Harmless in itself, and still worth closing, because it inverts the contract +// the check-syntax skill states: check is meant to be the strict gate and exec +// the thing that runs. Here exec was stricter, so a script could pass every +// pre-flight and then stop halfway through, having applied the statements before +// the typo and none after. +// +// The switch above this one already resolves the ALTER's MODULE. The document +// itself was the gap, which is why a misspelled module was reported and a +// misspelled document was not — a distinction with no meaning to the author. +// +// Scope. This covers the ALTER statements whose target is a document resolvable +// from a listing that already exists, and each one below was measured passing +// check and failing exec before it was added. It is not extended speculatively: +// a target this cannot resolve must be left alone rather than guessed at, since +// a false "not found" blocks a script that would have worked, which is worse +// than the silence it replaces. + +// validateAlterTarget reports an ALTER whose target document does not exist, +// counting documents the script itself creates earlier. +func validateAlterTarget(ctx *ExecContext, stmt ast.Statement, sc *scriptContext) error { + if !ctx.Connected() { + return nil + } + switch s := stmt.(type) { + case *ast.AlterPageStmt: + // One statement type, three document kinds — the visitor sets + // ContainerType from the keyword. + switch strings.ToUpper(s.ContainerType) { + case "SNIPPET": + return checkAlterTarget(ctx, sc, "snippet", s.PageName, + buildSnippetQualifiedNames, func(qn string) bool { return sc.snippets[qn] }) + case "LAYOUT": + return checkAlterTarget(ctx, sc, "layout", s.PageName, + buildLayoutQualifiedNames, func(qn string) bool { return sc.layouts[qn] }) + default: + return checkAlterTarget(ctx, sc, "page", s.PageName, + buildPageQualifiedNames, func(qn string) bool { return sc.pages[qn] }) + } + case *ast.AlterEntityStmt: + return checkAlterTarget(ctx, sc, "entity", s.Name, + buildEntityQualifiedNames, func(qn string) bool { return sc.entities[qn] }) + } + return nil +} + +// checkAlterTarget resolves one target, and on a miss reports the near names so +// the author can see the typo rather than only that they made one. +func checkAlterTarget(ctx *ExecContext, sc *scriptContext, kind string, name ast.QualifiedName, + known func(*ExecContext) map[string]bool, inScript func(string) bool) error { + qn := name.String() + // An unqualified name is a different error, already reported elsewhere; and + // a module the script creates has no listing to resolve against yet. + if name.Module == "" || sc.modules[name.Module] { + return nil + } + if inScript(qn) { + return nil + } + stored := known(ctx) + if stored[qn] { + return nil + } + // An empty listing means the backend could not answer, not that the project + // has no pages. Reporting "not found" from it would fail every script. + if len(stored) == 0 { + return nil + } + return mdlerrors.NewNotFoundMsg(kind, qn, fmt.Sprintf( + "%s not found: %s (referenced by alter %s)%s", kind, qn, kind, + nearNamesIn(stored, name.Module))) +} + +// nearNamesIn lists what the module does have, capped so a large module does not +// bury the error. The same shape as availableAttributes. +func nearNamesIn(stored map[string]bool, module string) string { + const max = 8 + var in []string + for qn := range stored { + if strings.HasPrefix(qn, module+".") { + in = append(in, strings.TrimPrefix(qn, module+".")) + } + } + if len(in) == 0 { + return "" + } + sort.Strings(in) + if len(in) > max { + return fmt.Sprintf(" — %s has %s and %d more", module, + strings.Join(in[:max], ", "), len(in)-max) + } + return fmt.Sprintf(" — %s has %s", module, strings.Join(in, ", ")) +} + +// buildLayoutQualifiedNames returns every layout qualified name in the project. +func buildLayoutQualifiedNames(ctx *ExecContext) map[string]bool { + result := make(map[string]bool) + h, err := getHierarchy(ctx) + if err != nil { + return result + } + layouts, err := ctx.Backend.ListLayouts() + if err != nil { + return result + } + for _, l := range layouts { + if l == nil { + continue + } + result[h.GetQualifiedName(l.ContainerID, l.Name)] = true + } + return result +} diff --git a/mdl/executor/validate_alter_target_test.go b/mdl/executor/validate_alter_target_test.go new file mode 100644 index 0000000000..0607300d44 --- /dev/null +++ b/mdl/executor/validate_alter_target_test.go @@ -0,0 +1,171 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// `alter page Mod.NoSuchPage` passed `mxcli check -p --references` and was then +// refused by exec with "page not found" — check was the weaker gate, which +// inverts the contract the check-syntax skill states and lets a script stop +// halfway through, with the statements before the typo already applied +// (ako/mxcli-rest FINDINGS #60). + +func alterTargetFixture(t *testing.T) *ExecContext { + t.Helper() + mod := &model.Module{Name: "Shop"} + mod.ID = nextID("mod") + + page := &pages.Page{Name: "Home_Web"} + page.ID = nextID("page") + page.ContainerID = mod.ID + + snippet := &pages.Snippet{Name: "Sn_Header"} + snippet.ID = nextID("snip") + snippet.ContainerID = mod.ID + + layout := &pages.Layout{Name: "App_Default"} + layout.ID = nextID("layout") + layout.ContainerID = mod.ID + + ent := &domainmodel.Entity{Name: "Order", Persistable: true} + ent.ID = nextID("ent") + dm := &domainmodel.DomainModel{ContainerID: mod.ID, Entities: []*domainmodel.Entity{ent}} + dm.ID = nextID("dm") + + h := mkHierarchy(mod) + withContainer(h, page.ID, mod.ID) + withContainer(h, snippet.ID, mod.ID) + withContainer(h, layout.ID, mod.ID) + withContainer(h, dm.ID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleByNameFunc: func(string) (*model.Module, error) { return mod, nil }, + ListPagesFunc: func() ([]*pages.Page, error) { return []*pages.Page{page}, nil }, + ListSnippetsFunc: func() ([]*pages.Snippet, error) { return []*pages.Snippet{snippet}, nil }, + ListLayoutsFunc: func() ([]*pages.Layout, error) { return []*pages.Layout{layout}, nil }, + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { return []*domainmodel.DomainModel{dm}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + return ctx +} + +func TestValidateAlterTarget_ReportsAMissingDocument(t *testing.T) { + ctx := alterTargetFixture(t) + + for _, tc := range []struct { + name string + stmt ast.Statement + want string // the kind named in the error + near string // a real name in the module, which the error should offer + }{ + {"page", &ast.AlterPageStmt{ContainerType: "PAGE", + PageName: ast.QualifiedName{Module: "Shop", Name: "Hoem_Web"}}, "page", "Home_Web"}, + {"snippet", &ast.AlterPageStmt{ContainerType: "SNIPPET", + PageName: ast.QualifiedName{Module: "Shop", Name: "Sn_Heade"}}, "snippet", "Sn_Header"}, + {"layout", &ast.AlterPageStmt{ContainerType: "LAYOUT", + PageName: ast.QualifiedName{Module: "Shop", Name: "App_Defualt"}}, "layout", "App_Default"}, + {"entity", &ast.AlterEntityStmt{ + Name: ast.QualifiedName{Module: "Shop", Name: "Ordr"}}, "entity", "Order"}, + } { + t.Run(tc.name, func(t *testing.T) { + err := validateAlterTarget(ctx, tc.stmt, newScriptContext()) + if err == nil { + t.Fatal("a misspelled target was accepted — exec would refuse it, so check is the weaker gate") + } + if !strings.Contains(err.Error(), tc.want+" not found") { + t.Errorf("the error should name the kind %q: %v", tc.want, err) + } + // A typo is much cheaper to fix when the error shows what is there. + if !strings.Contains(err.Error(), tc.near) { + t.Errorf("the error should offer %q as a near name: %v", tc.near, err) + } + }) + } +} + +// CONTROL: the targets that DO exist must pass. Without this a validator that +// always returned an error would satisfy the test above. +func TestValidateAlterTarget_AcceptsAnExistingDocument(t *testing.T) { + ctx := alterTargetFixture(t) + + for _, stmt := range []ast.Statement{ + &ast.AlterPageStmt{ContainerType: "PAGE", PageName: ast.QualifiedName{Module: "Shop", Name: "Home_Web"}}, + &ast.AlterPageStmt{ContainerType: "SNIPPET", PageName: ast.QualifiedName{Module: "Shop", Name: "Sn_Header"}}, + &ast.AlterPageStmt{ContainerType: "LAYOUT", PageName: ast.QualifiedName{Module: "Shop", Name: "App_Default"}}, + &ast.AlterEntityStmt{Name: ast.QualifiedName{Module: "Shop", Name: "Order"}}, + } { + if err := validateAlterTarget(ctx, stmt, newScriptContext()); err != nil { + t.Errorf("an existing target was rejected: %v", err) + } + } +} + +// CONTROL: a script that creates the document and then alters it is the common +// shape, and must not be broken by a check that only looks at the project. +func TestValidateAlterTarget_CountsDocumentsTheScriptCreates(t *testing.T) { + ctx := alterTargetFixture(t) + + sc := newScriptContext() + sc.pages["Shop.P_New"] = true + sc.snippets["Shop.Sn_New"] = true + sc.layouts["Shop.L_New"] = true + sc.entities["Shop.NewThing"] = true + + for _, stmt := range []ast.Statement{ + &ast.AlterPageStmt{ContainerType: "PAGE", PageName: ast.QualifiedName{Module: "Shop", Name: "P_New"}}, + &ast.AlterPageStmt{ContainerType: "SNIPPET", PageName: ast.QualifiedName{Module: "Shop", Name: "Sn_New"}}, + &ast.AlterPageStmt{ContainerType: "LAYOUT", PageName: ast.QualifiedName{Module: "Shop", Name: "L_New"}}, + &ast.AlterEntityStmt{Name: ast.QualifiedName{Module: "Shop", Name: "NewThing"}}, + } { + if err := validateAlterTarget(ctx, stmt, sc); err != nil { + t.Errorf("a document the script creates was reported missing: %v", err) + } + } +} + +// CONTROL: a module the script creates has no listing to resolve against, so +// its documents cannot be checked and must not be reported. +func TestValidateAlterTarget_SkipsAModuleTheScriptCreates(t *testing.T) { + ctx := alterTargetFixture(t) + sc := newScriptContext() + sc.modules["Fresh"] = true + + stmt := &ast.AlterPageStmt{ContainerType: "PAGE", + PageName: ast.QualifiedName{Module: "Fresh", Name: "Anything"}} + if err := validateAlterTarget(ctx, stmt, sc); err != nil { + t.Errorf("a document in a script-created module was reported missing: %v", err) + } +} + +// CONTROL: an empty listing means the backend could not answer, not that the +// project has no pages. Reporting "not found" from it would fail every script +// against a backend that does not implement the listing. +func TestValidateAlterTarget_SilentWhenTheListingIsEmpty(t *testing.T) { + mod := &model.Module{Name: "Shop"} + mod.ID = nextID("mod") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleByNameFunc: func(string) (*model.Module, error) { return mod, nil }, + ListPagesFunc: func() ([]*pages.Page, error) { return nil, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + stmt := &ast.AlterPageStmt{ContainerType: "PAGE", + PageName: ast.QualifiedName{Module: "Shop", Name: "Anything"}} + if err := validateAlterTarget(ctx, stmt, newScriptContext()); err != nil { + t.Errorf("an unanswerable listing produced a not-found: %v", err) + } +} diff --git a/mdl/executor/validate_create_shape.go b/mdl/executor/validate_create_shape.go new file mode 100644 index 0000000000..562aa7205b --- /dev/null +++ b/mdl/executor/validate_create_shape.go @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// Two statements that `mxcli check` accepted and `exec` or the build refused. +// Both are answerable from the statement alone, so they run in the no-project +// pass: a divergence that only `-p` catches is still a divergence. + +const ( + unqualifiedCreateRule = "MDL074" + voidReturnAliasRule = "MDL075" +) + +// ValidateCreateIsQualified reports a CREATE whose target name carries no +// module. +// +// `exec` refuses it — "module name is required: objects must be created within +// a module (use ModuleName.ObjectName syntax)" — and `check` passed it, so a +// script stopped partway through with the statements before it already applied +// and none after (mendixlabs/mxcli#1050). exec is not transactional, so "run it +// again" then hits "already exists" on the ones that did land. +// +// The rule is the whole reason MDL insists on qualified names: an unqualified +// one has no module to be created in, and there is no sensible default — the +// first module, the last one created, and "the only one" are all guesses that +// would put a document somewhere the author did not say. +// +// A MODULE is the exception and is excluded rather than special-cased at the +// call site: a module has nothing to be qualified BY. +func ValidateCreateIsQualified(prog *ast.Program) []linter.Violation { + if prog == nil { + return nil + } + var out []linter.Violation + for _, stmt := range prog.Statements { + if _, ok := stmt.(*ast.CreateModuleStmt); ok { + continue + } + docType, name, _ := stmtCreateInfo(stmt) + if docType == "" || name == "" || strings.Contains(name, ".") { + continue + } + out = append(out, linter.Violation{ + RuleID: unqualifiedCreateRule, + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s %q has no module — every document is created inside one, so `exec` "+ + "refuses this with \"module name is required\". Write Module.%s", + docType, name, name), + Location: linter.Location{DocumentType: docType, DocumentName: name}, + Suggestion: fmt.Sprintf( + "Qualify it: `Module.%s`. Reported here rather than at exec time because "+ + "exec is not transactional — the statements before this one are already "+ + "applied when it fails, and re-running then hits \"already exists\" on them.", + name), + }) + } + return out +} + +// ValidateVoidReturnAlias reports `RETURNS void AS $x`. +// +// The alias names the variable a flow returns, so pairing it with void is a +// contradiction — and mxcli believed the alias: it wrote `return $x` into a flow +// with no such variable, which the build rejects (mendixlabs/mxcli#1041): +// +// CREATE MICROFLOW … RETURNS void AS $result BEGIN COMMIT $Customer; END; +// mxcli check --references -> exit 0 +// mx check -> [CE0109] "Undefined variable 'result'." at End event +// +// Refused rather than repaired. Emitting a bare `return` would also build, but +// the two spellings mean different things to the author — one of them wrote an +// alias on purpose and meant to return something — and silently dropping it is +// how a flow comes to return nothing while its source still says otherwise. +// `RETURNS void` alone round-trips cleanly and is the fix when void was meant. +func ValidateVoidReturnAlias(prog *ast.Program) []linter.Violation { + if prog == nil { + return nil + } + var out []linter.Violation + for _, stmt := range prog.Statements { + var name, kind string + var rt *ast.MicroflowReturnType + switch s := stmt.(type) { + case *ast.CreateMicroflowStmt: + name, kind, rt = s.Name.String(), "microflow", s.ReturnType + case *ast.CreateNanoflowStmt: + name, kind, rt = s.Name.String(), "nanoflow", s.ReturnType + default: + continue + } + if rt == nil || rt.Variable == "" || !isVoidReturn(rt) { + continue + } + alias := strings.TrimPrefix(rt.Variable, "$") + out = append(out, linter.Violation{ + RuleID: voidReturnAliasRule, + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s %s is declared `RETURNS void AS $%s` — an alias names the variable the "+ + "flow returns, so it cannot be paired with void. mxcli writes `return $%s` "+ + "into a flow that has no such variable, which the build rejects as CE0109 "+ + "\"Undefined variable '%s'\"", + kind, name, alias, alias, alias), + Location: linter.Location{DocumentType: kind, DocumentName: name}, + Suggestion: fmt.Sprintf( + "Drop the alias (`RETURNS void`) if the flow returns nothing, or give it the "+ + "type $%s actually holds (`RETURNS AS $%s`).", alias, alias), + }) + } + return out +} + +// isVoidReturn reports whether a declared return type means "returns nothing". +// +// A nil clause is a flow with no RETURNS at all, which cannot carry an alias, so +// the caller's Variable check is what does the selecting. +func isVoidReturn(rt *ast.MicroflowReturnType) bool { + return rt == nil || rt.Type.Kind == ast.TypeVoid +} diff --git a/mdl/executor/validate_create_shape_test.go b/mdl/executor/validate_create_shape_test.go new file mode 100644 index 0000000000..ebd726b858 --- /dev/null +++ b/mdl/executor/validate_create_shape_test.go @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +func ruleFired(src, rule string) (bool, string) { + prog, errs := visitor.Build(src) + if len(errs) > 0 || prog == nil { + return false, "parse errors" + } + for _, v := range ValidateProgram(prog, "") { + if v.RuleID == rule { + return true, v.Message + } + } + return false, "" +} + +// `create association Order_Probe from …` passed `check --references` and was +// then refused by `exec` with "module name is required". exec is not +// transactional, so the statements before it were already applied and re-running +// hit "already exists" on them (mendixlabs/mxcli#1050). +func TestValidateCreateIsQualified(t *testing.T) { + fired, msg := ruleFired(`create association Order_Probe from A.Order to A.Customer type reference;`, "MDL074") + if !fired { + t.Fatal("an unqualified CREATE was accepted") + } + for _, want := range []string{"Order_Probe", "module"} { + if !strings.Contains(msg, want) { + t.Errorf("message should mention %q: %s", want, msg) + } + } + + // It is not association-specific: the same gap existed for every document. + for _, src := range []string{ + `create entity Thing ( Code: String(10) );`, + `create microflow ACT_Thing () begin log info 'x'; end;`, + `create enumeration Colours ( Red 'Red' );`, + } { + if ok, _ := ruleFired(src, "MDL074"); !ok { + t.Errorf("unqualified CREATE accepted: %s", src) + } + } +} + +// CONTROL: a qualified CREATE passes, and a MODULE — which has nothing to be +// qualified by — must never be reported. +func TestValidateCreateIsQualified_Controls(t *testing.T) { + for _, src := range []string{ + `create module MyModule;`, + `create entity A.Thing ( Code: String(10) );`, + `create association A.Order_Customer from A.Order to A.Customer type reference;`, + } { + if ok, msg := ruleFired(src, "MDL074"); ok { + t.Errorf("reported a valid statement: %s -> %s", src, msg) + } + } +} + +// `RETURNS void AS $result` passed check and wrote `return $result` into a flow +// with no such variable: [CE0109] "Undefined variable 'result'." at End event +// (mendixlabs/mxcli#1041). +func TestValidateVoidReturnAlias(t *testing.T) { + fired, msg := ruleFired( + `create microflow A.P () returns void as $result begin log info 'x'; end;`, "MDL075") + if !fired { + t.Fatal("`RETURNS void AS $result` was accepted") + } + for _, want := range []string{"result", "CE0109"} { + if !strings.Contains(msg, want) { + t.Errorf("message should mention %q: %s", want, msg) + } + } +} + +// CONTROL: the two spellings that are correct. `RETURNS void` alone is the fix +// the report itself identified as round-tripping cleanly, and a typed return +// with an alias is the ordinary shape — a rule that flagged either would break +// far more than it fixed. +func TestValidateVoidReturnAlias_Controls(t *testing.T) { + for _, src := range []string{ + `create microflow A.P () returns void begin log info 'x'; end;`, + `create microflow A.P () begin log info 'x'; end;`, + `create microflow A.P () returns string as $out begin set $out = 'x'; return $out; end;`, + } { + if ok, msg := ruleFired(src, "MDL075"); ok { + t.Errorf("reported a valid flow: %s -> %s", src, msg) + } + } +} + +// Both rules need NO project: a divergence that only `-p` catches is still a +// divergence, and these are answerable from the statement alone. +func TestCreateShapeRules_NeedNoProject(t *testing.T) { + prog, _ := visitor.Build(`create entity Thing ( Code: String(10) );`) + if len(ValidateCreateIsQualified(prog)) == 0 { + t.Error("MDL074 did not fire without a project") + } + prog, _ = visitor.Build(`create microflow A.P () returns void as $r begin log info 'x'; end;`) + if len(ValidateVoidReturnAlias(prog)) == 0 { + t.Error("MDL075 did not fire without a project") + } + // CONTROL: a nil program must not panic. + if len(ValidateCreateIsQualified(nil))+len(ValidateVoidReturnAlias(nil)) != 0 { + t.Error("a nil program produced violations") + } + _ = ast.Program{} +} diff --git a/mdl/executor/validate_dup_widget_test.go b/mdl/executor/validate_dup_widget_test.go index 1682a55164..164117aed6 100644 --- a/mdl/executor/validate_dup_widget_test.go +++ b/mdl/executor/validate_dup_widget_test.go @@ -17,7 +17,7 @@ func TestCheckDuplicateWidgetNames_Unit(t *testing.T) { {Type: "listview", Name: "ruTop"}, }}, } - errs := checkDuplicateWidgetNames(widgets) + errs := checkDuplicateWidgetNames(widgets, nil) if len(errs) != 1 || !strings.Contains(errs[0], "ruTop") { t.Fatalf("expected one duplicate error for ruTop, got %v", errs) } @@ -40,7 +40,7 @@ func TestCheckDuplicateWidgetNames_Parsed(t *testing.T) { if !ok { t.Fatalf("statement 0 = %T, want *ast.CreatePageStmtV3", prog.Statements[0]) } - dup := checkDuplicateWidgetNames(pg.Widgets) + dup := checkDuplicateWidgetNames(pg.Widgets, nil) if len(dup) != 1 || !strings.Contains(dup[0], "ruTop") { t.Fatalf("expected duplicate ruTop error from parsed page, got %v (widget names may not be populated for containers)", dup) } diff --git a/mdl/executor/validate_member_refs.go b/mdl/executor/validate_member_refs.go new file mode 100644 index 0000000000..5cfe57f64d --- /dev/null +++ b/mdl/executor/validate_member_refs.go @@ -0,0 +1,410 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// `check --references` resolved the DOCUMENT and the ENTITY a statement names, +// and then stopped at the door. A member name inside one — the attribute on the +// left of a CREATE / CHANGE assignment — was never resolved against the entity +// it belongs to, so a typo passed check, passed exec, and surfaced at the far +// end of a build as CE1613 (mendixlabs/mxcli#1048): +// +// CHANGE $Order ("IsArchived" = true); -- no such attribute +// mxcli check --references -> exit 0 +// mxcli exec -> "Created microflow" +// mx check -> [CE1613] "The selected attribute +// 'Bench.Order.IsArchived' no longer exists." +// +// exec does resolve the name — resolveAttributeInEntityHierarchy, in the flow +// builder — but when resolution FAILS it falls back to writing +// `.` anyway. That fabricated identifier is what mxbuild +// rejects. This check reports the same failure at the point it is cheap to fix. +// +// # Why this cannot simply mirror exec's resolver +// +// exec's resolver answers a two-valued question (resolved / did not resolve) +// because it has a fallback either way: it writes the fabricated name and lets +// the build complain. A CHECK has no such luxury — "did not resolve" conflates a +// real typo with a lookup that could not be PERFORMED, and reporting the second +// is a false error that blocks a script which would have worked. So the walk +// below has three outcomes and reports only the middle one. +// +// What that guard is and is not for, measured rather than assumed: +// +// - INHERITANCE is not the case. The walk follows GeneralizationRef through +// the backend, and the System module answers normally that way: +// System.FileDocument loads with its 6 attributes, so `CHANGE $File (Name = +// …)` on an entity extending it resolves, while a name on no link of the +// chain is correctly reported. (`update security` fails on System for an +// unrelated reason — it loads the domain-model UNIT BY ID out of +// mprcontents, where System has no file. Different mechanism, and not this +// one; mendixlabs/mxcli#1047.) +// +// - A BACKEND THAT CANNOT ANSWER is the case. mxcli has more than one +// (MPR, MCP/PED, mock), the interface permits an error from every lookup, +// and a check that turns "I could not look" into "your attribute is +// missing" fails every script against such a backend. That is what +// memberUnknown covers, and TestResolveMemberOnEntity_SilentWhenTheModelCannotBeRead +// is the control: flip that branch to memberMissing and it reports. +type memberResolution int + +const ( + // memberFound: the name is an attribute of the entity or of one of its + // generalizations. + memberFound memberResolution = iota + // memberMissing: the whole chain was walked to its root and the name is not + // there. Only this is reported. + memberMissing + // memberUnknown: a link in the chain could not be loaded, so the question + // was never answered. Silent. + memberUnknown +) + +// resolveMemberOnEntity walks entityQN and its generalizations looking for a +// member called memberName. +// +// Matching is case-sensitive, as Mendix's own is: mxbuild rejects `orderno` for +// `OrderNo` with the same CE1613, so accepting it here would let through the +// error this exists to catch. +func resolveMemberOnEntity(ctx *ExecContext, entityQN, memberName string) memberResolution { + b, ok := ctx.Backend.(entityLookupBackend) + if !ok || entityQN == "" || memberName == "" { + return memberUnknown + } + seen := map[string]bool{} + for currentQN := entityQN; currentQN != ""; { + if seen[currentQN] { + // A generalization cycle is not a model mxcli should reason about. + return memberUnknown + } + seen[currentQN] = true + + parts := strings.SplitN(currentQN, ".", 2) + if len(parts) != 2 { + return memberUnknown + } + mod, err := b.GetModuleByName(parts[0]) + if err != nil || mod == nil { + return memberUnknown + } + dm, err := b.GetDomainModel(mod.ID) + if err != nil || dm == nil { + // The System module lands here. Silence, not a report. + return memberUnknown + } + entity := dm.FindEntityByName(parts[1]) + if entity == nil { + return memberUnknown + } + for _, attr := range entity.Attributes { + if attr != nil && attr.Name == memberName { + return memberFound + } + } + // An association is a legal member of a CREATE / CHANGE too, and it is + // named on the entity that OWNS it rather than on either end, so it is + // checked against the module's whole association list rather than the + // entity's attributes. + if associationNamedInModule(dm, memberName) { + return memberFound + } + currentQN = entity.GeneralizationRef + } + return memberMissing +} + +// associationNamedInModule reports whether the domain model has an association +// of this bare name. A qualified member is handled by the caller; this covers +// the unqualified spelling, which resolves in the entity's own module. +func associationNamedInModule(dm *domainmodel.DomainModel, name string) bool { + for _, a := range dm.Associations { + if a != nil && a.Name == name { + return true + } + } + for _, a := range dm.CrossAssociations { + if a != nil && a.Name == name { + return true + } + } + return false +} + +// associationTargetFrom returns the entity at the other end of an association +// traversed from fromEntityQN. +// +// ParentID is the FROM entity (the foreign-key owner) and ChildID the TO entity +// — Mendix's inverted naming, per CLAUDE.md — and a retrieve may traverse from +// either end, so both directions resolve. +// +// A start entity that matches neither end returns false rather than guessing. +// That happens when the starting variable is a SPECIALISATION of the end, which +// this deliberately does not chase: the cost of being wrong is a false error on +// a working script, and the cost of being silent is one unchecked member. +func associationTargetFrom(ctx *ExecContext, assocQN, fromEntityQN string) (string, bool) { + if assocQN == "" || fromEntityQN == "" { + return "", false + } + b, ok := ctx.Backend.(entityLookupBackend) + if !ok { + return "", false + } + parts := strings.SplitN(assocQN, ".", 2) + if len(parts) != 2 { + return "", false + } + mod, err := b.GetModuleByName(parts[0]) + if err != nil || mod == nil { + return "", false + } + dm, err := b.GetDomainModel(mod.ID) + if err != nil || dm == nil { + return "", false + } + for _, a := range dm.Associations { + if a == nil || a.Name != parts[1] { + continue + } + from := entityQNByID(ctx, a.ParentID) + to := entityQNByID(ctx, a.ChildID) + switch fromEntityQN { + case from: + return to, to != "" + case to: + return from, from != "" + } + return "", false + } + return "", false +} + +// validateMemberReferences reports CREATE / CHANGE member names that do not +// exist on the entity they are assigned to. +// +// Runs in the --references pass: it needs the project, and the whole point is to +// answer a question `mxcli check` without -p cannot. +func validateMemberReferences(ctx *ExecContext, prog *ast.Program, sc *scriptContext) []string { + if prog == nil || !ctx.Connected() { + return nil + } + authored := authoredMembers(prog) + var out []string + for _, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.CreateMicroflowStmt: + out = append(out, checkFlowMembers(ctx, sc, authored, + s.Name.String(), s.Parameters, s.Body)...) + case *ast.CreateNanoflowStmt: + out = append(out, checkFlowMembers(ctx, sc, authored, + s.Name.String(), s.Parameters, s.Body)...) + } + } + return out +} + +// checkFlowMembers walks one flow body, tracking which entity each variable +// holds so a CHANGE can be resolved at all. +func checkFlowMembers(ctx *ExecContext, sc *scriptContext, authored map[string]bool, + flowQN string, params []ast.MicroflowParam, body []ast.MicroflowStatement) []string { + + vars := map[string]string{} + for _, p := range params { + if p.Name == "" { + continue + } + // A bare qualified name parses as TypeEnumeration with EnumRef set (see + // CLAUDE.md), so both spellings are recorded rather than guessed + // between. A name that is really an enumeration resolves no entity and + // is skipped, so the wrong guess costs nothing. + switch { + case p.Type.EntityRef != nil: + vars[p.Name] = p.Type.EntityRef.String() + case p.Type.Kind == ast.TypeEnumeration && p.Type.EnumRef != nil: + vars[p.Name] = p.Type.EnumRef.String() + } + } + + var out []string + var walk func(stmts []ast.MicroflowStatement) + walk = func(stmts []ast.MicroflowStatement) { + for _, st := range stmts { + switch s := st.(type) { + case *ast.CreateObjectStmt: + if s.Variable != "" { + vars[s.Variable] = s.EntityType.String() + } + out = append(out, checkMembers(ctx, sc, authored, flowQN, + fmt.Sprintf("create %s", s.EntityType.String()), + s.EntityType.String(), s.Changes)...) + case *ast.ChangeObjectStmt: + entityQN := vars[s.Variable] + out = append(out, checkMembers(ctx, sc, authored, flowQN, + fmt.Sprintf("change $%s", s.Variable), entityQN, s.Changes)...) + case *ast.RetrieveStmt: + // Source is the ENTITY for a database retrieve and the + // ASSOCIATION for an association retrieve (StartVariable set). + // Both are modelled: an association retrieve feeding a LOOP is + // the ordinary bulk-update shape, and leaving it untyped made + // the check silent on the commonest place a member is named. + switch { + case s.Variable == "" || s.Source.Name == "": + case s.StartVariable == "": + vars[s.Variable] = s.Source.String() + default: + if from, ok := vars[strings.TrimPrefix(s.StartVariable, "$")]; ok { + if to, ok := associationTargetFrom(ctx, s.Source.String(), from); ok { + vars[s.Variable] = to + } + } + } + case *ast.IfStmt: + walk(s.ThenBody) + walk(s.ElseBody) + case *ast.WhileStmt: + walk(s.Body) + case *ast.LoopStmt: + // The iterator holds the list's element type; without this a + // CHANGE on the loop variable is unresolvable and silent. + if s.LoopVariable != "" && s.ListVariable != "" { + if qn, ok := vars[s.ListVariable]; ok { + vars[s.LoopVariable] = qn + } + } + walk(s.Body) + } + if eh := stmtErrorHandling(st); eh != nil && len(eh.Body) > 0 { + walk(eh.Body) + } + } + } + walk(body) + return out +} + +// checkMembers resolves each member of one CREATE / CHANGE. +func checkMembers(ctx *ExecContext, sc *scriptContext, authored map[string]bool, + flowQN, where, entityQN string, changes []ast.ChangeItem) []string { + + // No entity established (an untyped variable, an enumeration parameter, a + // retrieve this walk did not model) — nothing to resolve against. + if entityQN == "" || !strings.Contains(entityQN, ".") { + return nil + } + // NOTE: there is deliberately no "skip entities the script creates" guard. + // It looks necessary and is not, and having it cost the check most of its + // reach: a self-contained script defines its own entities, so the guard + // skipped the whole file — including the example written to demonstrate + // this very check, which passed while exercising nothing. + // + // Two mechanisms already cover the case without the loss. An attribute the + // script DECLARES is in `authored`. An entity that is not in the project at + // all resolves to memberUnknown (FindEntityByName returns nil), which is + // silent. What is left over — an entity the project has, and a member + // neither it nor the script declares — is exactly the defect. + + var out []string + for _, ch := range changes { + name := strings.Trim(ch.Attribute, `"`) + if name == "" { + continue + } + // A qualified member is a different question with a different answer, + // and exec already refuses the one-qualifier form that cannot be an + // attribute (FINDINGS #51). Left alone here rather than second-guessed. + if strings.Contains(name, ".") { + continue + } + if authored[strings.ToLower(entityQN+"."+name)] { + continue + } + if resolveMemberOnEntity(ctx, entityQN, name) != memberMissing { + continue + } + out = append(out, fmt.Sprintf( + "%s: %s has no member %q (in %s)%s — mxbuild reports this as CE1613 "+ + "\"The selected attribute '%s.%s' no longer exists\"", + flowQN, entityQN, name, where, nearMembers(ctx, entityQN), entityQN, name)) + } + return out +} + +// authoredMembers collects every attribute the SCRIPT itself declares, so a +// script that adds an attribute and then assigns it is not reported. +// +// Without this the check fires on the ordinary shape of a migration script — +// add the column, then populate it — which is the change most likely to be +// written against an entity that does not yet have the member. +func authoredMembers(prog *ast.Program) map[string]bool { + out := map[string]bool{} + add := func(entityQN, attr string) { + if entityQN != "" && attr != "" { + out[strings.ToLower(entityQN+"."+strings.Trim(attr, `"`))] = true + } + } + for _, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.CreateEntityStmt: + for _, a := range s.Attributes { + add(s.Name.String(), a.Name) + } + case *ast.AlterEntityStmt: + if s.Operation == ast.AlterEntityAddAttribute && s.Attribute != nil { + add(s.Name.String(), s.Attribute.Name) + } + // A rename makes the NEW name legal for the rest of the script. + if s.Operation == ast.AlterEntityRenameAttribute { + add(s.Name.String(), s.NewName) + } + } + } + return out +} + +// nearMembers lists what the entity does have, capped so a wide entity does not +// bury the error. A typo is cheap to fix only when you can see what is there. +func nearMembers(ctx *ExecContext, entityQN string) string { + const max = 10 + b, ok := ctx.Backend.(entityLookupBackend) + if !ok { + return "" + } + parts := strings.SplitN(entityQN, ".", 2) + if len(parts) != 2 { + return "" + } + mod, err := b.GetModuleByName(parts[0]) + if err != nil || mod == nil { + return "" + } + dm, err := b.GetDomainModel(mod.ID) + if err != nil || dm == nil { + return "" + } + entity := dm.FindEntityByName(parts[1]) + if entity == nil { + return "" + } + var names []string + for _, a := range entity.Attributes { + if a != nil { + names = append(names, a.Name) + } + } + if len(names) == 0 { + return "" + } + sort.Strings(names) + if len(names) > max { + return fmt.Sprintf(" — it has %s and %d more", + strings.Join(names[:max], ", "), len(names)-max) + } + return fmt.Sprintf(" — it has %s", strings.Join(names, ", ")) +} diff --git a/mdl/executor/validate_member_refs_test.go b/mdl/executor/validate_member_refs_test.go new file mode 100644 index 0000000000..f77c977285 --- /dev/null +++ b/mdl/executor/validate_member_refs_test.go @@ -0,0 +1,365 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// `CHANGE $Order ("IsArchived" = true)` — no such attribute — passed +// `check --references`, passed `exec`, and surfaced as CE1613 at the far end of +// a build (mendixlabs/mxcli#1048). Reference checking resolved the entity and +// stopped there. + +// memberFixture builds Shop.Order (OrderNo, Status) extending Shop.Base (Code), +// plus an association, plus a second module whose domain model CANNOT be read — +// the backend-cannot-answer case the three-valued resolver exists for. +func memberFixture(t *testing.T) *ExecContext { + t.Helper() + shop := &model.Module{Name: "Shop"} + shop.ID = nextID("shop") + opaque := &model.Module{Name: "Opaque"} + opaque.ID = nextID("opaque") + + mkAttr := func(name string) *domainmodel.Attribute { + a := &domainmodel.Attribute{Name: name, Type: &domainmodel.StringAttributeType{}} + a.ID = nextID("attr" + name) + return a + } + + base := &domainmodel.Entity{Name: "Base", Persistable: true} + base.ID = nextID("base") + base.Attributes = []*domainmodel.Attribute{mkAttr("Code")} + + order := &domainmodel.Entity{Name: "Order", Persistable: true, GeneralizationRef: "Shop.Base"} + order.ID = nextID("order") + order.Attributes = []*domainmodel.Attribute{mkAttr("OrderNo"), mkAttr("Status")} + + customer := &domainmodel.Entity{Name: "Customer", Persistable: true} + customer.ID = nextID("cust") + customer.Attributes = []*domainmodel.Attribute{mkAttr("Name")} + + assoc := &domainmodel.Association{Name: "Order_Customer", ParentID: order.ID, ChildID: customer.ID} + assoc.ID = nextID("assoc") + + // An entity whose generalization lives in a module the backend refuses to + // answer for. Its own attributes resolve; anything else is unknowable. + imported := &domainmodel.Entity{Name: "Imported", Persistable: true, GeneralizationRef: "Opaque.Thing"} + imported.ID = nextID("imported") + imported.Attributes = []*domainmodel.Attribute{mkAttr("LocalOnly")} + + dm := &domainmodel.DomainModel{ContainerID: shop.ID, + Entities: []*domainmodel.Entity{base, order, customer, imported}, + Associations: []*domainmodel.Association{assoc}, + } + dm.ID = nextID("dm") + + h := mkHierarchy(shop) + withContainer(h, dm.ID, shop.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{shop, opaque}, nil }, + GetModuleByNameFunc: func(name string) (*model.Module, error) { + switch name { + case "Shop": + return shop, nil + case "Opaque": + return opaque, nil + } + return nil, fmt.Errorf("no module %q", name) + }, + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { + return []*domainmodel.DomainModel{dm}, nil + }, + GetDomainModelFunc: func(id model.ID) (*domainmodel.DomainModel, error) { + if id == shop.ID { + return dm, nil + } + return nil, fmt.Errorf("domain model %s cannot be read", id) + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + return ctx +} + +func changeStmt(variable string, attrs ...string) *ast.ChangeObjectStmt { + s := &ast.ChangeObjectStmt{Variable: variable} + for _, a := range attrs { + s.Changes = append(s.Changes, ast.ChangeItem{Attribute: a}) + } + return s +} + +func flowProgram(params []ast.MicroflowParam, body ...ast.MicroflowStatement) *ast.Program { + return &ast.Program{Statements: []ast.Statement{ + &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "Shop", Name: "ACT_Probe"}, + Parameters: params, + Body: body, + }, + }} +} + +func orderParam() []ast.MicroflowParam { + return []ast.MicroflowParam{{ + Name: "Order", + Type: ast.DataType{EntityRef: &ast.QualifiedName{Module: "Shop", Name: "Order"}}, + }} +} + +func TestValidateMemberReferences_ReportsAMissingAttribute(t *testing.T) { + ctx := memberFixture(t) + msgs := validateMemberReferences(ctx, + flowProgram(orderParam(), changeStmt("Order", `"IsArchived"`)), newScriptContext()) + + if len(msgs) != 1 { + t.Fatalf("got %d messages, want 1: %v", len(msgs), msgs) + } + for _, want := range []string{"Shop.Order", "IsArchived", "CE1613"} { + if !strings.Contains(msgs[0], want) { + t.Errorf("message should mention %q: %s", want, msgs[0]) + } + } + // A typo is cheap to fix only when the error shows what is there. + if !strings.Contains(msgs[0], "OrderNo") { + t.Errorf("message should offer the entity's real members: %s", msgs[0]) + } +} + +// CONTROL: everything that legitimately resolves must stay silent. Each of +// these is a shape a real script uses, and each would be a false error that +// blocks a script which builds cleanly. +func TestValidateMemberReferences_AcceptsWhatResolves(t *testing.T) { + ctx := memberFixture(t) + + for _, tc := range []struct { + name string + prog *ast.Program + sc *scriptContext + }{ + { + name: "an attribute the entity declares", + prog: flowProgram(orderParam(), changeStmt("Order", `"Status"`)), + sc: newScriptContext(), + }, + { + // The walk has to follow GeneralizationRef, or every specialisation + // reports its inherited members as missing. + name: "an attribute INHERITED from the generalization", + prog: flowProgram(orderParam(), changeStmt("Order", `"Code"`)), + sc: newScriptContext(), + }, + { + // An association is a legal member of a CREATE/CHANGE, and it is not + // in the entity's attribute list. + name: "an association as a member", + prog: flowProgram(orderParam(), changeStmt("Order", "Order_Customer")), + sc: newScriptContext(), + }, + { + // exec already refuses the one-qualifier form that cannot be an + // attribute (FINDINGS #51); this check leaves qualified names alone + // rather than second-guessing that. + name: "a qualified member", + prog: flowProgram(orderParam(), changeStmt("Order", "Shop.Order_Customer")), + sc: newScriptContext(), + }, + { + name: "a variable this walk cannot type", + prog: flowProgram(nil, changeStmt("Mystery", `"Whatever"`)), + sc: newScriptContext(), + }, + } { + t.Run(tc.name, func(t *testing.T) { + if msgs := validateMemberReferences(ctx, tc.prog, tc.sc); len(msgs) != 0 { + t.Errorf("reported a member that resolves: %v", msgs) + } + }) + } +} + +// The migration shape — add the column, then populate it — is the change most +// likely to name a member the STORED project does not have yet. Reporting it +// would break the scripts this check is supposed to help. +func TestValidateMemberReferences_CountsMembersTheScriptAuthors(t *testing.T) { + ctx := memberFixture(t) + + prog := &ast.Program{Statements: []ast.Statement{ + &ast.AlterEntityStmt{ + Name: ast.QualifiedName{Module: "Shop", Name: "Order"}, + Operation: ast.AlterEntityAddAttribute, + Attribute: &ast.Attribute{Name: "Archived"}, + }, + &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "Shop", Name: "ACT_Probe"}, + Parameters: orderParam(), + Body: []ast.MicroflowStatement{changeStmt("Order", `"Archived"`)}, + }, + }} + if msgs := validateMemberReferences(ctx, prog, newScriptContext()); len(msgs) != 0 { + t.Errorf("reported an attribute the script adds earlier: %v", msgs) + } + + // CONTROL: the guard is scoped to what the script actually authors — a + // different name on the same entity is still reported. + prog.Statements[1].(*ast.CreateMicroflowStmt).Body = + []ast.MicroflowStatement{changeStmt("Order", `"Archivd"`)} + if msgs := validateMemberReferences(ctx, prog, newScriptContext()); len(msgs) != 1 { + t.Errorf("a typo beside an authored attribute went unreported: %v", msgs) + } +} + +// An entity the script creates is not in the stored project, so the member is +// unknowable rather than missing — and that falls out of the resolver rather +// than needing a guard of its own. An explicit "skip entities the script +// creates" guard used to sit here and cost the check most of its reach: a +// self-contained script defines its own entities, so the guard skipped the whole +// file, including the example written to demonstrate the check. +func TestValidateMemberReferences_SkipsAnEntityTheScriptCreates(t *testing.T) { + ctx := memberFixture(t) + sc := newScriptContext() + sc.entities["Shop.Fresh"] = true + + prog := flowProgram(nil, &ast.CreateObjectStmt{ + Variable: "F", + EntityType: ast.QualifiedName{Module: "Shop", Name: "Fresh"}, + Changes: []ast.ChangeItem{{Attribute: `"Anything"`}}, + }) + if msgs := validateMemberReferences(ctx, prog, sc); len(msgs) != 0 { + t.Errorf("reported a member on an entity the script creates: %v", msgs) + } + if got := resolveMemberOnEntity(ctx, "Shop.Fresh", "Anything"); got != memberUnknown { + t.Errorf("an entity absent from the project = %v, want memberUnknown", got) + } + + // CONTROL: an entity the script creates that DOES already exist in the + // project is checked normally — the script declaring it is not a licence to + // stop resolving its members. + prog2 := flowProgram(nil, &ast.CreateObjectStmt{ + Variable: "O", + EntityType: ast.QualifiedName{Module: "Shop", Name: "Order"}, + Changes: []ast.ChangeItem{{Attribute: `"Nonsense"`}}, + }) + sc2 := newScriptContext() + sc2.entities["Shop.Order"] = true + if msgs := validateMemberReferences(ctx, prog2, sc2); len(msgs) != 1 { + t.Errorf("a stored entity restated by the script went unchecked: %v", msgs) + } +} + +// THE CONTROL the package comment names. A backend that cannot answer must +// produce silence, not a report — mxcli has more than one backend and every +// lookup in the interface may return an error. Shop.Imported's generalization +// lives in a module whose domain model the fixture refuses to read, so a name +// that is not among Imported's own attributes is unknowable, not missing. +// +// Collapse memberUnknown into memberMissing (a two-valued resolver, which is +// what exec uses) and this reports. +func TestResolveMemberOnEntity_SilentWhenTheModelCannotBeRead(t *testing.T) { + ctx := memberFixture(t) + + if got := resolveMemberOnEntity(ctx, "Shop.Imported", "LocalOnly"); got != memberFound { + t.Errorf("an attribute the entity itself declares = %v, want memberFound", got) + } + if got := resolveMemberOnEntity(ctx, "Shop.Imported", "CouldBeInherited"); got != memberUnknown { + t.Errorf("a name past an unreadable generalization = %v, want memberUnknown", got) + } + // And nothing is reported for it. + prog := flowProgram([]ast.MicroflowParam{{ + Name: "Imp", + Type: ast.DataType{EntityRef: &ast.QualifiedName{Module: "Shop", Name: "Imported"}}, + }}, changeStmt("Imp", `"CouldBeInherited"`)) + if msgs := validateMemberReferences(ctx, prog, newScriptContext()); len(msgs) != 0 { + t.Errorf("reported a member the backend could not rule out: %v", msgs) + } +} + +// The full chain must be walked before anything is reported: a name on neither +// the entity nor its generalization is the real defect. +func TestResolveMemberOnEntity_WalksTheWholeChain(t *testing.T) { + ctx := memberFixture(t) + if got := resolveMemberOnEntity(ctx, "Shop.Order", "Code"); got != memberFound { + t.Errorf("inherited attribute = %v, want memberFound", got) + } + if got := resolveMemberOnEntity(ctx, "Shop.Order", "Nonsense"); got != memberMissing { + t.Errorf("absent attribute = %v, want memberMissing", got) + } +} + +// A CHANGE on a loop iterator resolves through the list it iterates; without +// that the commonest bulk-update shape goes unchecked. +func TestValidateMemberReferences_TypesALoopIterator(t *testing.T) { + ctx := memberFixture(t) + prog := flowProgram(nil, + &ast.RetrieveStmt{Variable: "Orders", + Source: ast.QualifiedName{Module: "Shop", Name: "Order"}}, + &ast.LoopStmt{LoopVariable: "O", ListVariable: "Orders", + Body: []ast.MicroflowStatement{changeStmt("O", `"Bogus"`)}}, + ) + msgs := validateMemberReferences(ctx, prog, newScriptContext()) + if len(msgs) != 1 || !strings.Contains(msgs[0], "Bogus") { + t.Errorf("a CHANGE on a loop iterator went unchecked: %v", msgs) + } +} + +// An association retrieve feeding a LOOP is the ordinary bulk-update shape, and +// leaving it untyped made the check silent on the commonest place a member is +// named. Both directions resolve, because a retrieve may traverse from either +// end (ParentID is the FROM entity, ChildID the TO — CLAUDE.md). +func TestValidateMemberReferences_TypesAnAssociationRetrieve(t *testing.T) { + ctx := memberFixture(t) + + // From the Customer end: Order_Customer traversed from Customer gives Orders. + prog := flowProgram([]ast.MicroflowParam{{ + Name: "Cust", + Type: ast.DataType{EntityRef: &ast.QualifiedName{Module: "Shop", Name: "Customer"}}, + }}, + &ast.RetrieveStmt{Variable: "Orders", StartVariable: "Cust", + Source: ast.QualifiedName{Module: "Shop", Name: "Order_Customer"}}, + &ast.LoopStmt{LoopVariable: "O", ListVariable: "Orders", + Body: []ast.MicroflowStatement{changeStmt("O", `"Bogus"`)}}, + ) + msgs := validateMemberReferences(ctx, prog, newScriptContext()) + if len(msgs) != 1 || !strings.Contains(msgs[0], "Shop.Order") { + t.Errorf("an association retrieve from the TO end went unchecked: %v", msgs) + } + + // CONTROL: a real member of the traversed-to entity stays silent, so the + // direction was resolved rather than merely producing some entity. + prog.Statements[0].(*ast.CreateMicroflowStmt).Body[1].(*ast.LoopStmt).Body = + []ast.MicroflowStatement{changeStmt("O", `"OrderNo"`)} + if msgs := validateMemberReferences(ctx, prog, newScriptContext()); len(msgs) != 0 { + t.Errorf("resolved the wrong end of the association: %v", msgs) + } +} + +// CONTROL for the coverage boundary. A variable bound by an activity this walk +// does not model — `send rest request`, `response: file as`, an association +// retrieve from an untyped start — stays untyped, and an untyped target is +// UNCHECKED rather than wrongly checked. Measured on ako/mxcli-rest: every +// CHANGE in its 14 scripts is bound that way, so the check fires there only on +// CREATE OBJECT (which names its entity and is therefore always typed). +// +// This is a real limit, not a bug: widening it means typing more binders, and +// each one is only safe once its output entity is known for certain. +func TestValidateMemberReferences_UntypedTargetIsUnchecked(t *testing.T) { + ctx := memberFixture(t) + prog := flowProgram(nil, + // StartVariable is never bound, so the retrieve yields nothing typed. + &ast.RetrieveStmt{Variable: "Kids", StartVariable: "FromNowhere", + Source: ast.QualifiedName{Module: "Shop", Name: "Order_Customer"}}, + &ast.LoopStmt{LoopVariable: "K", ListVariable: "Kids", + Body: []ast.MicroflowStatement{changeStmt("K", `"DefinitelyNotAMember"`)}}, + ) + if msgs := validateMemberReferences(ctx, prog, newScriptContext()); len(msgs) != 0 { + t.Errorf("checked a member against an entity it could not establish: %v", msgs) + } +} diff --git a/mdl/executor/validate_navigation_icons.go b/mdl/executor/validate_navigation_icons.go new file mode 100644 index 0000000000..e5d2fdceb8 --- /dev/null +++ b/mdl/executor/validate_navigation_icons.go @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// validateMenuItemIcons (MDL074) flags a navigation menu item that specifies no +// icon. +// +// # Why this is worth a warning +// +// Mendix's navigation sidebar collapses to an icon rail, and that is the state +// most users leave it in. A collapsed item shows its icon; an item without one +// falls back to the first few characters of its caption, which is rarely enough +// to tell "Orders" from "Order lines". The menu still builds, `mx check` passes, +// and the only symptom is a column of truncated words in a browser. +// +// The icon is optional in the grammar and nothing said anything about leaving it +// out, so an iconless menu was the easiest one to write. +// +// # Every item, at every depth +// +// A sub-item is a menu item: it renders in the flyout the collapsed rail opens, +// and a submenu's PARENT sits directly on the rail, so it needs one most of all. +// The rule is uniform rather than scoped to the top level — if that proves noisy +// on a deep menu, narrowing it is a one-line change to the walk, but reporting +// too little is the failure that is hard to notice. +// +// # A warning, not an error +// +// The project builds and runs. This is a usability defect, not a broken model, +// and `exec` refuses only on errors — a rule that blocked the script would make +// an opinion about design into a gate. +// +// # It needs no project +// +// The icon's PRESENCE is in the script; only its target has to be resolved +// against the project's icon collections, and MDL-ICON01 already does that +// separately. So this runs in the project-free pass and fires under +// `mxcli check` with no `-p` — which is how CI runs it, and how the four widget +// rules that were inert in CI got missed. +func validateMenuItemIcons(stmt ast.Statement) []linter.Violation { + var items []ast.NavMenuItemDef + var where string + + switch s := stmt.(type) { + case *ast.AlterNavigationStmt: + items, where = s.MenuItems, "navigation "+s.ProfileName + case *ast.CreateMenuStmt: + items, where = s.Items, "menu "+s.Name.String() + default: + return nil + } + + var out []linter.Violation + var walk func(list []ast.NavMenuItemDef) + walk = func(list []ast.NavMenuItemDef) { + for _, item := range list { + // NOT `item.Icon == ""`. A glyph icon carries a numeric code and no + // name, so the obvious test reports an item that plainly has an + // icon — and every Studio Pro-authored menu is full of them. Ask the + // kind, which is what the writer and DESCRIBE also read. + if item.IconKind == types.MenuIconNone { + out = append(out, linter.Violation{ + RuleID: "MDL074", + Severity: linter.SeverityWarning, + Message: fmt.Sprintf( + "%s: menu item %q specifies no icon — a collapsed navigation sidebar "+ + "shows only the icon, so this item appears as a few characters of its caption", + where, item.Caption), + Suggestion: "add `icon Atlas_Core.Atlas.` (list them with " + + "`describe icon collection Atlas_Core.Atlas`)", + }) + } + walk(item.Items) + } + } + walk(items) + return out +} diff --git a/mdl/executor/validate_navigation_icons_test.go b/mdl/executor/validate_navigation_icons_test.go new file mode 100644 index 0000000000..ee57c3e82c --- /dev/null +++ b/mdl/executor/validate_navigation_icons_test.go @@ -0,0 +1,163 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/types" +) + +func navIconWarnings(vs []linter.Violation) []linter.Violation { + var out []linter.Violation + for _, v := range vs { + if v.RuleID == "MDL074" { + out = append(out, v) + } + } + return out +} + +// A menu item with no icon is flagged. +// +// Mendix's navigation sidebar collapses to an icon rail. A collapsed item shows +// its icon; an item without one shows the first few characters of its caption, +// which is rarely enough to tell "Orders" from "Order lines" — so the menu is +// unusable in exactly the state most users leave it in. +// +// The icon is optional in the grammar and nothing said anything, so an item +// written without one built cleanly, passed `mx check`, and only looked wrong in +// a browser. +func TestValidateMenuItemIcons_FlagsMissingIcon(t *testing.T) { + stmt := &ast.AlterNavigationStmt{ + ProfileName: "Responsive", + MenuItems: []ast.NavMenuItemDef{ + {Caption: "Home", Icon: "Atlas_Core.Atlas.home", IconKind: types.MenuIconCollection}, + {Caption: "Orders"}, + }, + } + + got := navIconWarnings(validateMenuItemIcons(stmt)) + if len(got) != 1 { + t.Fatalf("got %d warnings, want 1: %+v", len(got), got) + } + if !strings.Contains(got[0].Message, "Orders") { + t.Errorf("warning does not name the item: %q", got[0].Message) + } + if got[0].Severity != linter.SeverityWarning { + t.Errorf("severity = %v, want warning — a menu without icons builds and runs, "+ + "it is just hard to use collapsed", got[0].Severity) + } +} + +// The control: an item WITH an icon must stay silent, or the rule is "warn on +// every menu item" and the first assertion proves nothing. +func TestValidateMenuItemIcons_IconedItemIsSilent(t *testing.T) { + stmt := &ast.AlterNavigationStmt{ + ProfileName: "Responsive", + MenuItems: []ast.NavMenuItemDef{ + {Caption: "Home", Icon: "Atlas_Core.Atlas.home", IconKind: types.MenuIconCollection}, + {Caption: "Admin", Icon: `Atlas_Core.Atlas."align-center"`, IconKind: types.MenuIconCollection, Items: []ast.NavMenuItemDef{ + {Caption: "Users", Icon: "Atlas_Core.Atlas.user", IconKind: types.MenuIconCollection}, + }}, + }, + } + if got := navIconWarnings(validateMenuItemIcons(stmt)); len(got) != 0 { + t.Errorf("a fully iconed menu produced %d warnings: %+v", len(got), got) + } +} + +// Sub-items are menu items too. A submenu's children render in the flyout the +// collapsed rail opens, and the parent itself sits ON the rail. +func TestValidateMenuItemIcons_RecursesIntoSubItems(t *testing.T) { + stmt := &ast.AlterNavigationStmt{ + ProfileName: "Responsive", + MenuItems: []ast.NavMenuItemDef{ + {Caption: "Admin", Icon: "Atlas_Core.Atlas.cog", IconKind: types.MenuIconCollection, Items: []ast.NavMenuItemDef{ + {Caption: "Users"}, + {Caption: "Roles", Icon: "Atlas_Core.Atlas.group", IconKind: types.MenuIconCollection}, + {Caption: "Deep", Items: []ast.NavMenuItemDef{{Caption: "Deeper"}}}, + }}, + }, + } + got := navIconWarnings(validateMenuItemIcons(stmt)) + if len(got) != 3 { + t.Fatalf("got %d warnings, want 3 (Users, Deep, Deeper): %+v", len(got), got) + } + joined := "" + for _, v := range got { + joined += v.Message + "\n" + } + for _, want := range []string{"Users", "Deep", "Deeper"} { + if !strings.Contains(joined, want) { + t.Errorf("%q not reported; recursion stops short:\n%s", want, joined) + } + } + if strings.Contains(joined, "Roles") { + t.Errorf("an iconed sub-item was reported:\n%s", joined) + } +} + +// A standalone menu document carries the same items, so it gets the same rule — +// CREATE MENU and CREATE NAVIGATION share NavMenuItemDef precisely so the two +// cannot diverge. +func TestValidateMenuItemIcons_CoversMenuDocuments(t *testing.T) { + stmt := &ast.CreateMenuStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Main_Menu"}, + Items: []ast.NavMenuItemDef{{Caption: "Reports"}}, + } + got := navIconWarnings(validateMenuItemIcons(stmt)) + if len(got) != 1 { + t.Fatalf("got %d warnings, want 1 for a menu document: %+v", len(got), got) + } + if !strings.Contains(got[0].Message, "MyModule.Main_Menu") { + t.Errorf("warning does not name the document: %q", got[0].Message) + } +} + +// Nothing to say about a statement with no menu at all, or another statement +// type entirely. +func TestValidateMenuItemIcons_Quiet(t *testing.T) { + if got := validateMenuItemIcons(&ast.AlterNavigationStmt{ProfileName: "Responsive"}); len(got) != 0 { + t.Errorf("a profile with no MENU block warned: %+v", got) + } + if got := validateMenuItemIcons(&ast.CreateEntityStmt{}); len(got) != 0 { + t.Errorf("an unrelated statement warned: %+v", got) + } +} + +// The rule needs no project, so it must fire under `mxcli check` with none — +// which is how CI runs it. A rule that only works with -p is inert in CI, the +// trap that hid four defects in the widget work. +func TestValidateMenuItemIcons_RunsWithoutAProject(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.AlterNavigationStmt{ + ProfileName: "Responsive", + MenuItems: []ast.NavMenuItemDef{{Caption: "Orders"}}, + }, + }} + if got := navIconWarnings(ValidateProgram(prog, "")); len(got) != 1 { + t.Errorf("got %d MDL074 from ValidateProgram with no project, want 1: %+v", len(got), got) + } +} + +// The trap this rule walked straight into: a glyph icon has a numeric code and +// NO name, so a check written as `Icon == ""` reports an item that plainly has +// an icon. Every Studio Pro-authored menu is full of them — the fixture's own +// Home item carries glyph 57377 — so the naive rule warned about the majority of +// real menus while claiming they had no icon. +func TestValidateMenuItemIcons_AGlyphIconIsAnIcon(t *testing.T) { + stmt := &ast.AlterNavigationStmt{ + ProfileName: "Responsive", + MenuItems: []ast.NavMenuItemDef{ + {Caption: "Home", IconKind: types.MenuIconGlyph, IconCode: 57377}, + {Caption: "Logo", IconKind: types.MenuIconImage, Icon: "MyMod.Images.logo"}, + }, + } + if got := navIconWarnings(validateMenuItemIcons(stmt)); len(got) != 0 { + t.Errorf("warned about items that have icons: %+v", got) + } +} diff --git a/mdl/executor/validate_page_context.go b/mdl/executor/validate_page_context.go index 34a9010cff..46ec9969cb 100644 --- a/mdl/executor/validate_page_context.go +++ b/mdl/executor/validate_page_context.go @@ -16,7 +16,7 @@ import ( // // This runs at check time (no MPR needed) and catches issues that would otherwise // only surface as CE errors in Studio Pro. -func validatePageContextTree(params []ast.PageParameter, widgets []*ast.WidgetV3) []string { +func validatePageContextTree(ctx *ExecContext, params []ast.PageParameter, widgets []*ast.WidgetV3) []string { // Build param name set paramNames := make(map[string]bool, len(params)) for _, p := range params { @@ -29,11 +29,21 @@ func validatePageContextTree(params []ast.PageParameter, widgets []*ast.WidgetV3 // Walk the widget tree with context tracking var errors []string - errors = append(errors, checkDuplicateWidgetNames(widgets)...) + errors = append(errors, checkDuplicateWidgetNames(widgets, pageContextWidgetRegistry(ctx))...) walkWidgetsWithContext(widgets, paramNames, widgetNames, false, &errors) return errors } +// pageContextWidgetRegistry returns the widget registry for the run, or nil. +// The duplicate-name rule needs it to tell an object-list item from a widget; +// every other check here is registry-free, and a nil ctx (unit tests) is fine. +func pageContextWidgetRegistry(ctx *ExecContext) *WidgetRegistry { + if ctx == nil { + return nil + } + return ctx.GetWidgetRegistry() +} + // widgetKindsWithoutStoredNames are the widget kinds Mendix stores with no Name // at all, so MDL's name for one is mxcli's own — derived at describe time to give // ALTER PAGE something to address. @@ -51,29 +61,70 @@ var widgetKindsWithoutStoredNames = map[string]bool{ "column": true, } +// objectListContainerKinds returns the lowercased MDL container keywords the +// widget's definition declares as object lists (a BarChart's `series`, an +// Accordion's `group`). Children with one of those types are ITEMS, not widgets. +// +// Returns nil for anything that does not resolve — a built-in widget, an unknown +// name, or no registry at all — so the caller's behaviour is unchanged there. +func objectListContainerKinds(registry *WidgetRegistry, w *ast.WidgetV3) map[string]bool { + if registry == nil || w == nil { + return nil + } + def := lookupWidgetDef(w, registry) + if def == nil { + return nil + } + out := make(map[string]bool, len(def.ObjectLists)) + for _, ol := range def.ObjectLists { + if ol.MDLContainer != "" { + out[strings.ToLower(ol.MDLContainer)] = true + } + } + // WidgetMode carries no ObjectLists — object lists are declared once on the + // definition — so there is nothing mode-scoped to add here. + return out +} + // checkDuplicateWidgetNames flags any widget name that appears more than once on a // page. Mendix requires widget names to be unique per page and rejects duplicates // with CE0495 "Duplicate name" — which mxcli check otherwise passed (FINDINGS #15). // Each duplicate name is reported once, in first-seen order. // // Widget kinds Mendix stores without a name are skipped: see -// widgetKindsWithoutStoredNames. -func checkDuplicateWidgetNames(widgets []*ast.WidgetV3) []string { +// widgetKindsWithoutStoredNames. So are OBJECT-LIST ITEMS — a chart `series`, a +// gallery `customitem` — for the same reason and with the same evidence: a page +// authored with `series sRegion (…)` comes back from DESCRIBE as +// `series series1 (…)`, because the stored WidgetObject carries no name and +// DESCRIBE has to synthesise one. Measured on mxbuild 11.6.6, three charts on +// one page each holding a `series s` is 0 errors; mxcli called it CE0495 and, +// since a reference error fails the run, refused to execute the script. +// +// registry may be nil (check runs with no project in CI). Then no parent +// resolves, itemKinds is empty everywhere, and the rule behaves as it did +// before — a false positive is preferable to a silent one, and the enumerated +// widget types are unaffected either way. +func checkDuplicateWidgetNames(widgets []*ast.WidgetV3, registry *WidgetRegistry) []string { counts := make(map[string]int) var order []string - var walk func(ws []*ast.WidgetV3) - walk = func(ws []*ast.WidgetV3) { + // itemKinds are the container keywords the ENCLOSING widget declares as + // object lists. Read from that widget's definition rather than from a list + // of keywords: the containers are def-driven, so a table here would be the + // second list this proposal exists to remove. + var walk func(ws []*ast.WidgetV3, itemKinds map[string]bool) + walk = func(ws []*ast.WidgetV3, itemKinds map[string]bool) { for _, w := range ws { - if w.Name != "" && !widgetKindsWithoutStoredNames[strings.ToLower(w.Type)] { + kind := strings.ToLower(w.Type) + if w.Name != "" && !widgetKindsWithoutStoredNames[kind] && !itemKinds[kind] { if counts[w.Name] == 0 { order = append(order, w.Name) } counts[w.Name]++ } - walk(w.Children) + walk(w.Children, objectListContainerKinds(registry, w)) } } - walk(widgets) + walk(widgets, nil) var errors []string for _, name := range order { diff --git a/mdl/executor/validate_page_context_objectlist_test.go b/mdl/executor/validate_page_context_objectlist_test.go new file mode 100644 index 0000000000..bae7d2688e --- /dev/null +++ b/mdl/executor/validate_page_context_objectlist_test.go @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// chartRegistry is a registry holding one definition that declares a `series` +// object list — the shape a real chart's .def.json has. Built in memory because +// NO embedded widget definition declares an object list (measured: 0 of them), +// so a test using LoadWidgetRegistry("") would skip everywhere, and the .def.json +// cache a project builds from its .mpk files is gitignored, so a test reading one +// would skip in CI. +func chartRegistry() *WidgetRegistry { + def := &WidgetDefinition{ + WidgetID: "com.mendix.widget.web.barchart.BarChart", + MDLName: "barchart", + ObjectLists: []ObjectListMapping{ + {PropertyKey: "series", MDLContainer: "SERIES"}, + }, + } + return &WidgetRegistry{ + byMDLName: map[string]*WidgetDefinition{"BARCHART": def}, + byWidgetID: map[string]*WidgetDefinition{def.WidgetID: def}, + } +} + +// dashboardWithThreeSeries mirrors the shape 34-chart-widget-examples.mdl uses: +// three separate charts on one page, each with a series the author called `s`. +func dashboardWithThreeSeries() []*ast.WidgetV3 { + chart := func(name, container, itemName string) *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "pluggablewidget", Name: name, + Properties: map[string]any{"WidgetType": "com.mendix.widget.web.barchart.BarChart"}, + Children: []*ast.WidgetV3{{Type: container, Name: itemName}}, + } + } + return []*ast.WidgetV3{ + chart("dashBar", "series", "s"), + chart("dashColumn", "series", "s"), + chart("dashArea", "series", "s"), + } +} + +// An object-list ITEM's name is mxcli's own, not the model's — the same reason +// widgetKindsWithoutStoredNames already excludes rows and columns. +// +// Proof the model does not hold it: a page authored with `series sRegion (…)` +// comes back from DESCRIBE as `series series1 (…)`. DESCRIBE synthesises the +// name because the stored WidgetObject has none, so no two of them can collide +// under CE0495. +// +// Proof mxbuild agrees: the page above, exec'd into a real 11.6.6 project and +// run through `mx check`, reports 0 errors. The control that the check is not +// inert on that project: a view entity with a deliberately wrong column type in +// the same app fails CE6770, so mxbuild was really validating. +// +// mxcli reported `duplicate widget name 's' (used 3 times)` and, because a +// reference error fails the run, refused to execute the script at all. +func TestCheckDuplicateWidgetNames_ObjectListItemsAreNotWidgets(t *testing.T) { + got := checkDuplicateWidgetNames(dashboardWithThreeSeries(), chartRegistry()) + for _, e := range got { + if strings.Contains(e, "'s'") { + t.Errorf("object-list item names counted as widget names: %q", e) + } + } +} + +// The control: real duplicate WIDGET names must still be reported. A fix that +// stopped descending into a pluggable widget's children — or that skipped every +// child of one — would pass the test above and turn CE0495 detection off for +// everything inside a chart or a gallery. +func TestCheckDuplicateWidgetNames_RealDuplicatesStillReported(t *testing.T) { + widgets := []*ast.WidgetV3{ + {Type: "dynamictext", Name: "dup"}, + {Type: "container", Name: "c", Children: []*ast.WidgetV3{ + {Type: "dynamictext", Name: "dup"}, + }}, + } + got := checkDuplicateWidgetNames(widgets, chartRegistry()) + if len(got) != 1 || !strings.Contains(got[0], "'dup'") { + t.Errorf("a genuine duplicate widget name was not reported: %v", got) + } +} + +// The second control: a child of a pluggable widget that is NOT one of its +// object-list containers is a real widget in a child slot, and two of those +// sharing a name IS CE0495. +func TestCheckDuplicateWidgetNames_ChildSlotWidgetsStillCount(t *testing.T) { + widgets := []*ast.WidgetV3{ + {Type: "pluggablewidget", Name: "w1", + Properties: map[string]any{"WidgetType": "com.mendix.widget.web.barchart.BarChart"}, + Children: []*ast.WidgetV3{{Type: "dynamictext", Name: "dup"}}}, + {Type: "dynamictext", Name: "dup"}, + } + got := checkDuplicateWidgetNames(widgets, chartRegistry()) + if len(got) != 1 || !strings.Contains(got[0], "'dup'") { + t.Errorf("a duplicate inside a pluggable widget's child slot was not reported: %v", got) + } +} + +// The third control: without a registry nothing can be resolved, and the rule +// must fall back to its previous behaviour rather than silently accepting +// everything. `check` runs with no project in CI, so this is the common path. +func TestCheckDuplicateWidgetNames_NoRegistryStillReports(t *testing.T) { + widgets := []*ast.WidgetV3{ + {Type: "dynamictext", Name: "dup"}, + {Type: "dynamictext", Name: "dup"}, + } + if got := checkDuplicateWidgetNames(widgets, nil); len(got) != 1 { + t.Errorf("with no registry, a plain duplicate must still be reported: %v", got) + } +} diff --git a/mdl/executor/validate_page_context_test.go b/mdl/executor/validate_page_context_test.go index 5dfee86b5b..230662ad54 100644 --- a/mdl/executor/validate_page_context_test.go +++ b/mdl/executor/validate_page_context_test.go @@ -25,7 +25,7 @@ func TestValidatePageContextTree_ParameterDSValid(t *testing.T) { }, } - errors := validatePageContextTree(params, widgets) + errors := validatePageContextTree(nil, params, widgets) if len(errors) > 0 { t.Errorf("Expected no errors, got: %v", errors) } @@ -44,7 +44,7 @@ func TestValidatePageContextTree_ParameterDSInvalid(t *testing.T) { }, } - errors := validatePageContextTree(params, widgets) + errors := validatePageContextTree(nil, params, widgets) if len(errors) != 1 { t.Fatalf("Expected 1 error, got %d: %v", len(errors), errors) } @@ -72,7 +72,7 @@ func TestValidatePageContextTree_SelectionDSValid(t *testing.T) { }, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) > 0 { t.Errorf("Expected no errors, got: %v", errors) } @@ -88,7 +88,7 @@ func TestValidatePageContextTree_SelectionDSInvalid(t *testing.T) { }, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) != 1 { t.Fatalf("Expected 1 error, got %d: %v", len(errors), errors) } @@ -102,7 +102,7 @@ func TestValidatePageContextTree_AttributeWithoutContext(t *testing.T) { {Type: "textbox", Name: "txtName", Properties: map[string]any{"Attribute": "Name"}}, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) != 1 { t.Fatalf("Expected 1 error, got %d: %v", len(errors), errors) } @@ -125,14 +125,14 @@ func TestValidatePageContextTree_AttributeInsideDataView(t *testing.T) { }, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) > 0 { t.Errorf("Expected no errors, got: %v", errors) } } func TestValidatePageContextTree_NoErrors(t *testing.T) { - errors := validatePageContextTree(nil, nil) + errors := validatePageContextTree(nil, nil, nil) if len(errors) > 0 { t.Errorf("Expected no errors for nil widgets, got: %v", errors) } diff --git a/mdl/executor/validate_page_unnamed_widgets_test.go b/mdl/executor/validate_page_unnamed_widgets_test.go index 4d7dc743c8..77c3f49bbf 100644 --- a/mdl/executor/validate_page_unnamed_widgets_test.go +++ b/mdl/executor/validate_page_unnamed_widgets_test.go @@ -57,7 +57,7 @@ func TestCheckDuplicateWidgetNames_IgnoresUnnamedWidgetKinds(t *testing.T) { ), } - if errs := checkDuplicateWidgetNames(page); len(errs) != 0 { + if errs := checkDuplicateWidgetNames(page, nil); len(errs) != 0 { t.Errorf("a widget kind whose name is not stored cannot be a CE0495 duplicate; got:\n %s", strings.Join(errs, "\n ")) } @@ -77,7 +77,7 @@ func TestCheckDuplicateWidgetNames_StillCatchesRealDuplicates(t *testing.T) { ), } - errs := checkDuplicateWidgetNames(page) + errs := checkDuplicateWidgetNames(page, nil) if len(errs) != 1 { t.Fatalf("got %d errors, want 1:\n %s", len(errs), strings.Join(errs, "\n ")) } @@ -95,7 +95,7 @@ func TestCheckDuplicateWidgetNames_NamedWidgetCollidingWithADerivedName(t *testi namedWidget("container", "row1"), } - errs := checkDuplicateWidgetNames(page) + errs := checkDuplicateWidgetNames(page, nil) if len(errs) != 1 || !strings.Contains(errs[0], "row1") { t.Fatalf("two containers named row1 are a real duplicate; got %d errors:\n %s", len(errs), strings.Join(errs, "\n ")) diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 263500cd27..2602635032 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -52,6 +52,10 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { if roleStmt, ok := stmt.(*ast.CreateUserRoleStmt); ok { violations = append(violations, ValidateUserRoleSystemModuleRole(roleStmt, securityEnabled)...) } + // A navigation menu item with no icon is unreadable once the sidebar is + // collapsed to its icon rail (MDL074). Covers both statements that carry + // menu items, which share one AST node so they cannot diverge. + violations = append(violations, validateMenuItemIcons(stmt)...) // A page with parameters and a Url must name each parameter in it (CE5601). if pageStmt, ok := stmt.(*ast.CreatePageStmtV3); ok { violations = append(violations, ValidatePageURLParameters(pageStmt)...) @@ -199,6 +203,25 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // (#927). violations = append(violations, ValidateExportMappingMembers(prog)...) + // Flag a template parameter written as a `$variable/` path where the + // position takes an attribute of the widget's context object. The writer + // keeps the prefix as part of the attribute name, so the model names an + // attribute that cannot exist (mendixlabs/mxcli#1046). The answer is in the + // statement, so it runs here rather than under --references — otherwise + // `mxcli check page.mdl` would stay silent on a mistake it can see. + violations = append(violations, ValidateWidgetParamPaths(prog)...) + + // Flag a CREATE whose target name has no module. `exec` refuses it and + // `check` passed it, so a script stopped partway through with the earlier + // statements already applied (mendixlabs/mxcli#1050). + violations = append(violations, ValidateCreateIsQualified(prog)...) + + // Flag `RETURNS void AS $x`. The alias names the returned variable, so + // pairing it with void is a contradiction — and mxcli believed the alias, + // writing `return $x` into a flow with no such variable (CE0109, + // mendixlabs/mxcli#1041). + violations = append(violations, ValidateVoidReturnAlias(prog)...) + // Flag a CUSTOM NAME MAP entry that matches nothing in the snippet. Silence // there made a typo indistinguishable from not writing the entry, which is // how #272's missing `item of` stayed hidden (ako/mxcli#272). diff --git a/mdl/executor/validate_widget_aliases_test.go b/mdl/executor/validate_widget_aliases_test.go new file mode 100644 index 0000000000..2ae48f412e --- /dev/null +++ b/mdl/executor/validate_widget_aliases_test.go @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// A property mapping's MdlAliases are the names a person is meant to WRITE. +// They must be accepted by the property validator, or the validator rejects the +// only spelling the documentation offers. +// +// # The defect +// +// A PieChart binds its data at the widget level, so its def.json carries +// +// {"propertyKey": "seriesValueAttribute", "source": "Attribute", +// "operation": "attribute", "mdlAliases": ["ValueAttribute"]} +// +// and the builder resolves `ValueAttribute:` through that alias — measured: the +// stored page comes back from DESCRIBE as `seriesValueAttribute: Total`, so the +// value persists. allowedWidgetProperties built its set from PropertyKey and +// Source only, so `ValueAttribute` was unknown and MDL-WIDGET01 fired. Since +// exec refuses to run a script with errors, the false positive did not merely +// warn — it blocked the page from being written at all. +// +// This is the "two lists, nothing comparing them" class again, and the sibling +// list in widget_defs.go (`mapped`, for knownProperties) already walks +// MdlAliases — so the two disagreed about the same def.json. +func TestAllowedWidgetProperties_IncludesMdlAliases(t *testing.T) { + def := &WidgetDefinition{ + WidgetID: "com.acme.Test", + MDLName: "test", + PropertyMappings: []PropertyMapping{ + {PropertyKey: "seriesValueAttribute", Source: "Attribute", + Operation: "attribute", MdlAliases: []string{"ValueAttribute"}}, + }, + } + + allowed, keys := allowedWidgetProperties(def) + + if !allowed["valueattribute"] { + t.Errorf("the alias `ValueAttribute` is not an allowed property; allowed keys: %v", keys) + } + // The control: the schema key must STILL be allowed. A fix that swapped one + // name for the other would pass the assertion above and break every script + // written against the schema key — including DESCRIBE's own output, which + // emits `seriesValueAttribute`. + if !allowed["seriesvalueattribute"] { + t.Errorf("the schema key `seriesValueAttribute` stopped being allowed; allowed keys: %v", keys) + } + // The suggestion list is what "did you mean" reads, so a typo'd alias should + // point back at the alias, not only at the schema key. + var sawAlias bool + for _, k := range keys { + if k == "ValueAttribute" { + sawAlias = true + } + } + if !sawAlias { + t.Errorf("`ValueAttribute` missing from the suggestion list %v — a typo would be told to "+ + "use the internal name instead of the documented one", keys) + } +} + +// Mode-scoped mappings carry aliases too, and go through the same helper. +// +// The alias here is deliberately NOT a case variant of the property key. The +// PieChart's real pair is `seriesName` / `SeriesName`, which collapses to one +// entry once lowercased — so a test using it passes against the broken code and +// proves nothing. +func TestAllowedWidgetProperties_IncludesMdlAliasesInModes(t *testing.T) { + def := &WidgetDefinition{ + WidgetID: "com.acme.Test", + MDLName: "test", + Modes: []WidgetMode{{ + PropertyMappings: []PropertyMapping{ + {PropertyKey: "seriesSortAttribute", Source: "Attribute", + Operation: "attribute", MdlAliases: []string{"SortAttribute"}}, + }, + }}, + } + allowed, keys := allowedWidgetProperties(def) + if !allowed["sortattribute"] { + t.Errorf("mode-scoped alias `SortAttribute` not allowed; keys: %v", keys) + } +} + +// The end-to-end shape of the bug, against the real embedded/installed +// definitions rather than a hand-built one: a PieChart written the documented +// way must not produce MDL-WIDGET01. +// +// Skips when the PieChart definition is not available (it ships in Charts.mpk, +// not in the embedded set), so this is a bonus assertion — the two above are the +// ones that always run. +func TestPieChartValueAttributeIsAccepted(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Skip("no registry") + } + def, ok := registry.GetByWidgetID("com.mendix.widget.web.piechart.PieChart") + if !ok || def == nil { + t.Skip("PieChart definition not available without a project") + } + allowed, keys := allowedWidgetProperties(def) + if !allowed["valueattribute"] { + t.Errorf("PieChart rejects `ValueAttribute:`, the name propertyAliases registers "+ + "and the builder resolves; allowed: %s", strings.Join(keys, ", ")) + } +} diff --git a/mdl/executor/validate_widget_hidden.go b/mdl/executor/validate_widget_hidden.go index fc40e14a98..5d86fbf328 100644 --- a/mdl/executor/validate_widget_hidden.go +++ b/mdl/executor/validate_widget_hidden.go @@ -158,6 +158,20 @@ func registryProjectPath(registry *WidgetRegistry) string { return registry.projectPath } +// sharedSourceOffMode reports whether an item property's shared Source lookup +// must be skipped because the item's dataSet mode selects its sibling. +// +// Scoped to a chart series' datasource pair, which is the one place two item +// properties claim the same Source. Anything wider would blind the +// hidden-property rule on every other object list, so the gate mirrors the +// builder's own condition rather than generalising it. +func sharedSourceOffMode(mapping *ObjectListMapping, m ItemPropertyMapping, dataSetMode string) bool { + if m.Operation != "datasource" || !isChartSeriesContainer(mapping.MDLContainer) { + return false + } + return !seriesDataSourceMatchesMode(m.PropertyKey, dataSetMode) +} + // itemValueMap resolves an object-list item's sub-property values (keyed by // lowercased schema key) and reports which the MDL set explicitly. The item form // of widgetValueMap. @@ -165,10 +179,24 @@ func itemValueMap(item *ast.WidgetV3, mapping *ObjectListMapping) (values map[st values = map[string]string{} explicit = map[string]bool{} + // A chart series' two datasource sub-properties SHARE the Source name + // "DataSource" (measured on linechart.def.json: staticDataSource and + // dynamicDataSource both declare it, neither declares an alias), so the + // mode selects which one the friendly `DataSource:` lands in. The builder + // routes on it — buildObjectListItem consults "DataSource" only for the + // property seriesDataSourceMatchesMode picks — so the checker has to as + // well, or it reports a property the script never wrote. + itemDataSetMode := "static" + if v, ok := lookupProperty(item.Properties, "dataSet"); ok { + if s := stringifyAny(v); s != "" { + itemDataSetMode = s + } + } + for _, m := range mapping.ItemProperties { key := strings.ToLower(m.PropertyKey) val, set := "", false - if m.Source != "" { + if m.Source != "" && !sharedSourceOffMode(mapping, m, itemDataSetMode) { if v, ok := lookupWidgetProp(item, m.Source); ok { val, set = v, true } diff --git a/mdl/executor/validate_widget_hidden_dataset_test.go b/mdl/executor/validate_widget_hidden_dataset_test.go new file mode 100644 index 0000000000..9ebfebfdcc --- /dev/null +++ b/mdl/executor/validate_widget_hidden_dataset_test.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// chartSeriesItem builds a chart `series`/`line` item written the documented way: +// one friendly `DataSource:` plus the dataSet mode that selects which schema +// property it lands in. +func chartSeriesItem(dataSet string) *ast.WidgetV3 { + return &ast.WidgetV3{ + Name: "s1", + Properties: map[string]any{ + "DataSet": dataSet, + "DataSource": &ast.DataSourceV3{Type: "database", Reference: "Mod.View"}, + }, + } +} + +// chartSeriesMapping mirrors the shape a chart's def.json actually has: BOTH +// datasource sub-properties declare Source "DataSource" and neither declares an +// alias, so a lookup by Source alone cannot tell them apart. +func chartSeriesMapping() *ObjectListMapping { + return &ObjectListMapping{ + MDLContainer: "SERIES", + PropertyKey: "series", + ItemProperties: []ItemPropertyMapping{ + {PropertyKey: "dataSet", Operation: "primitive", Value: "static"}, + {PropertyKey: "staticDataSource", Source: "DataSource", Operation: "datasource"}, + {PropertyKey: "dynamicDataSource", Source: "DataSource", Operation: "datasource"}, + }, + } +} + +// The friendly `DataSource:` on a chart series is routed BY dataSet mode when the +// item is built: buildObjectListItem looks the alias up only for the property +// seriesDataSourceMatchesMode selects, so `dataSet: 'static'` writes +// staticDataSource and leaves dynamicDataSource unset. +// +// itemValueMap resolved it by Source instead, which both properties share, so it +// reported dynamicDataSource as explicitly set — and MDL-WIDGET10 then warned +// that a value the script never wrote "will be ignored". Measured: 11 such +// warnings on 34-chart-widget-examples.mdl, one per series in the file, on the +// only syntax the examples and skills document. +func TestItemValueMap_FriendlyDataSourceIsModeRouted(t *testing.T) { + _, explicit := itemValueMap(chartSeriesItem("static"), chartSeriesMapping()) + + if !explicit["staticdatasource"] { + t.Error("staticDataSource not explicit under dataSet 'static' — the mode-matching " + + "property is the one the builder writes, so the checker must see it set") + } + if explicit["dynamicdatasource"] { + t.Error("dynamicDataSource reported as explicitly set under dataSet 'static'; " + + "buildObjectListItem never writes it, so MDL-WIDGET10 warns about a value " + + "that does not exist") + } +} + +// The control: the routing must follow the mode rather than always preferring +// `static`. A fix that hardcoded "ignore dynamic*" would pass the test above and +// break every dynamic series. +func TestItemValueMap_FriendlyDataSourceFollowsDynamicMode(t *testing.T) { + _, explicit := itemValueMap(chartSeriesItem("dynamic"), chartSeriesMapping()) + + if !explicit["dynamicdatasource"] { + t.Error("dynamicDataSource not explicit under dataSet 'dynamic' — this is the " + + "property the builder writes in that mode") + } + if explicit["staticdatasource"] { + t.Error("staticDataSource reported as explicitly set under dataSet 'dynamic'") + } +} + +// The second control: the gate is scoped to chart series. A non-chart object list +// whose sub-properties share a Source must keep resolving by Source, or the +// hidden-property rule goes blind on every other widget. +func TestItemValueMap_NonChartContainerStillResolvesBySource(t *testing.T) { + mapping := &ObjectListMapping{ + MDLContainer: "COLUMN", + PropertyKey: "columns", + ItemProperties: []ItemPropertyMapping{ + {PropertyKey: "attribute", Source: "Attribute", Operation: "attribute"}, + }, + } + item := &ast.WidgetV3{Name: "c1", Properties: map[string]any{"Attribute": "Name"}} + + _, explicit := itemValueMap(item, mapping) + if !explicit["attribute"] { + t.Error("a non-chart item property stopped resolving through its Source") + } +} + +// The third control: an item that names a datasource by its SCHEMA key keeps +// working regardless of mode. Someone writing `dynamicDataSource:` explicitly +// means it, and the builder honours it (the PropertyKey lookup runs before the +// mode-aware fallback), so the checker must see it set. +func TestItemValueMap_ExplicitSchemaKeyIgnoresMode(t *testing.T) { + item := &ast.WidgetV3{ + Name: "s1", + Properties: map[string]any{ + "DataSet": "static", + "dynamicDataSource": &ast.DataSourceV3{Type: "database", Reference: "Mod.View"}, + }, + } + _, explicit := itemValueMap(item, chartSeriesMapping()) + if !explicit["dynamicdatasource"] { + t.Error("an explicitly named dynamicDataSource was dropped by the mode gate — " + + "the gate applies to the shared Source lookup, not to the schema key") + } +} diff --git a/mdl/executor/validate_widget_kind.go b/mdl/executor/validate_widget_kind.go new file mode 100644 index 0000000000..1949167472 --- /dev/null +++ b/mdl/executor/validate_widget_kind.go @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "path/filepath" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/sdk/widgets/mpk" +) + +// Slice 0 of PROPOSAL_def_driven_widget_bodies.md: teach the validator what a +// widget is. +// +// Until now the GRAMMAR was the widget-kind validator — `widgetTypeV3` is an +// allow-list, so an unknown kind could not parse and the validator never needed +// an independent notion of one. Two mistakes already slip past that, because +// neither is a keyword the parser checks, and both reached `exec` with `check` +// reporting success: +// +// pluggablewidget 'com.acme.NotAWidget' w1 -- the id is a string literal +// group g1 (…) inside HTML Element -- a real keyword, wrong parent +// +// Reporting them is worth doing on its own. It is also what makes the +// def-driven body (slices 2-3) safe: those give up the parser's enforcement, so +// the semantic check has to exist first. + +// validateWidgetKind reports a widget whose kind mxcli cannot resolve, and a +// container keyword the parent's definition does not declare. +func validateWidgetKind(w *ast.WidgetV3, registry *WidgetRegistry, parentDef *WidgetDefinition, + parentObjectLists map[string]*ObjectListMapping, locationPrefix string) []linter.Violation { + if w == nil || registry == nil { + return nil + } + + // An explicit widget id that resolves to nothing. Only reachable through the + // `pluggablewidget ''` / `customwidget ''` forms, where the id is a + // string literal the parser cannot check. + if id, ok := w.Properties["WidgetType"].(string); ok && id != "" { + // With no project there is nothing to be unknown RELATIVE TO: the + // registry holds only the embedded widgets, so every real project + // widget would be reported. `mxcli check` with no -p is the common + // case — it is how the example corpus is checked in CI — and without + // this one example file alone produced 14 violations. + // + // Scoped to this branch on purpose: the container rule below needs a + // resolvable PARENT, not a project, and works from a definition alone. + if registryProjectPath(registry) == "" { + return nil + } + if _, known := registry.GetByWidgetID(id); !known && !packageInstalledFor(registry, id) { + return []linter.Violation{{ + RuleID: "MDL-WIDGET25", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: widget `%s` has no definition for %q%s", + locationPrefix, w.Name, id, nearestWidgetIDs(registry, id)), + Suggestion: "install the widget from the Marketplace so its .mpk lands in widgets/, or check the id for a typo", + }} + } + return nil + } + + // A generic widget type — one the grammar accepted as a bare IDENTIFIER + // rather than as an enumerated widget token (slice 2). It can only be a + // widget definition's MDL name, so failing to resolve one is a mistake, + // not a built-in mxcli happens not to know. + // + // Without this the typo `htmlelemnt` reaches validateStaticWidgetUnknownProps + // and is reported as an unrecognized PROPERTY (MDL-WIDGET07, a warning), so + // `check` exits 0 having complained about the wrong thing. Measured before + // this branch existed: `htmlelemnt frame (tagName: 'div')` gave + // "0 errors, 1 warning" about `tagName`, while the correct spelling was + // completely clean. + // + // Same project requirement as the id branch below, and for the same reason: + // with no project the registry holds only the embedded widgets, so every + // real one would be reported. + if w.TypeIsGeneric && registryProjectPath(registry) != "" { + if _, known := registry.Get(strings.ToUpper(w.Type)); known { + return nil + } + // A container the parent declares is not a widget and never resolves in + // the registry — that is what makes it a container. Accept it here and + // let the object-list engine validate its properties. + if parentDef != nil && + (parentObjectLists[strings.ToUpper(w.Type)] != nil || parentDeclaresSlot(parentDef, w.Type)) { + return nil + } + // Inside a resolvable parent, "not a container of " beats "not a + // widget": it names what the parent DOES declare, which is the answer + // the author needs. `attribut` inside an htmlelement is a misspelt + // container, not a missing widget package. + if parentDef != nil { + return []linter.Violation{{ + RuleID: "MDL-WIDGET26", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` is not a container of %s%s", + locationPrefix, strings.ToLower(w.Type), parentLabel(parentDef), declaredContainers(parentDef)), + Suggestion: "use one of the parent widget's own containers, or move this out of the widget's body", + }} + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET25", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` is not a widget in this project%s", + locationPrefix, strings.ToLower(w.Type), nearestWidgetNames(registry, w.Type)), + Suggestion: "run `mxcli widget init` if the package was just installed, or `describe widget ` to see what is available", + }} + } + + // A container keyword used where the parent does not declare it. These + // keywords mean nothing on their own — `group` is not a widget — so one + // outside a parent that declares it can only be a mistake. + if !isObjectListContainerKeyword(w.Type, registry) { + return nil + } + // Never judge a container against a parent that cannot be resolved. In a + // project that has not run `widget init` the registry knows only the + // embedded widgets, so every real parent looks container-less and every + // container would be reported. Silence is the honest answer there. + if parentDef == nil { + return nil + } + if parentObjectLists[strings.ToUpper(w.Type)] != nil || parentDeclaresSlot(parentDef, w.Type) { + return nil + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET26", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` is not a container of %s%s", + locationPrefix, strings.ToLower(w.Type), parentLabel(parentDef), declaredContainers(parentDef)), + Suggestion: "use one of the parent widget's own containers, or move this out of the widget's body", + }} +} + +// isObjectListContainerKeyword reports whether a keyword is an object-list +// container rather than a widget in its own right. +// +// Derived from the registry — the union of every known definition's container +// keywords — rather than restated. A hardcoded list here was already wrong: +// isUniversalObjectListKeyword names seven where the grammar has nine, missing +// SCALECOLOR, CUSTOMBUTTON and ALLOWEDFILEFORMAT, which is the same list-drift +// this proposal exists to remove. +func isObjectListContainerKeyword(widgetType string, registry *WidgetRegistry) bool { + if widgetType == "" || registry == nil { + return false + } + up := strings.ToUpper(widgetType) + // A keyword that names a widget is a widget, whatever else it may be. + if _, isWidget := registry.Get(up); isWidget { + return false + } + for _, def := range registry.All() { + for _, ol := range def.ObjectLists { + if strings.EqualFold(ol.MDLContainer, widgetType) { + return true + } + } + } + return isUniversalObjectListKeyword(widgetType) +} + +func parentLabel(parentDef *WidgetDefinition) string { + if parentDef == nil { + return "this widget" + } + if parentDef.MDLName != "" { + return "`" + strings.ToLower(parentDef.MDLName) + "`" + } + return "`" + parentDef.WidgetID + "`" +} + +// declaredContainers names what the parent DOES declare, so the reader is not +// left to guess. A parent with none says so, which is the more useful answer. +func declaredContainers(parentDef *WidgetDefinition) string { + if parentDef == nil { + return "" + } + var names []string + for _, ol := range parentDef.ObjectLists { + names = append(names, strings.ToLower(ol.MDLContainer)) + } + for _, cs := range parentDef.ChildSlots { + names = append(names, strings.ToLower(cs.MDLContainer)) + } + if len(names) == 0 { + return ", which declares no containers" + } + sort.Strings(names) + return " — it declares: " + strings.Join(names, ", ") +} + +// nearestWidgetIDs suggests known ids sharing the unknown one's last segment, +// which is where a typo or a wrong vendor prefix usually shows. +func nearestWidgetIDs(registry *WidgetRegistry, id string) string { + last := id + if i := strings.LastIndex(id, "."); i >= 0 { + last = id[i+1:] + } + var hits []string + for _, def := range registry.All() { + if strings.EqualFold(def.WidgetID, id) { + continue + } + if strings.Contains(strings.ToLower(def.WidgetID), strings.ToLower(last)) { + hits = append(hits, def.WidgetID) + } + } + if len(hits) == 0 { + return "" + } + sort.Strings(hits) + if len(hits) > 3 { + hits = hits[:3] + } + return " — did you mean " + strings.Join(hits, ", ") + "?" +} + +// packageInstalledFor reports whether the project has a widget package for this +// id, even though no definition has been extracted from it yet. +// +// Load-bearing against a false-positive storm. LoadWidgetRegistry reads only +// `.mxcli/widgets/*.def.json`; unlike the page builder's registry it does NOT +// refresh those from installed .mpk files. So in a project that has never run +// `widget init`, the validator knows the nine embedded widgets and nothing else +// — and calling every real project widget "unknown" would be worse than the +// silence this rule replaces. +// +// Asking whether the package is installed is the same question slice 1's error +// message asks, and it separates "mxcli has not looked at this widget yet" from +// "this widget does not exist". +func packageInstalledFor(registry *WidgetRegistry, widgetID string) bool { + dir := registryProjectPath(registry) + if dir == "" { + return false + } + found, err := mpk.FindMPK(filepath.Dir(dir), widgetID) + return err == nil && found != "" +} + +// parentDeclaresSlot reports whether the parent declares a CHILD SLOT by this +// name. Object lists alone are not the whole vocabulary of a widget body, and +// treating a declared slot as undeclared would report correct MDL. +func parentDeclaresSlot(parentDef *WidgetDefinition, keyword string) bool { + if parentDef == nil { + return false + } + for _, cs := range parentDef.ChildSlots { + if strings.EqualFold(cs.MDLContainer, keyword) { + return true + } + } + return false +} + +// nearestWidgetNames suggests known MDL names close to an unresolved one, +// using the same edit-distance helper MDL-WIDGET07 uses for property keys. +// +// A hand-rolled prefix heuristic was tried first and is not good enough: the +// commonest real mistake is a dropped letter in the MIDDLE (`htmlelemnt`), +// which shares no useful prefix with `htmlelement` past the typo, so it +// suggested nothing at all on the very case that motivated the rule. +func nearestWidgetNames(registry *WidgetRegistry, name string) string { + if registry == nil || name == "" { + return "" + } + var candidates []string + for _, def := range registry.All() { + if def.MDLName != "" { + candidates = append(candidates, strings.ToLower(def.MDLName)) + } + } + sort.Strings(candidates) + if best := nearestKey(name, candidates); best != "" { + return " — did you mean `" + best + "`?" + } + return "" +} diff --git a/mdl/executor/validate_widget_kind_test.go b/mdl/executor/validate_widget_kind_test.go new file mode 100644 index 0000000000..b8a227df2b --- /dev/null +++ b/mdl/executor/validate_widget_kind_test.go @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// widgetKindViolations runs the widget-tree validator over one page body. +func widgetKindViolations(t *testing.T, projectPath string, widgets []*ast.WidgetV3) []string { + t.Helper() + registry := LoadWidgetRegistry(projectPath) + if registry == nil { + t.Fatal("no widget registry") + } + var out []string + for _, v := range validateWidgetTree(widgets, registry, "page X") { + out = append(out, v.RuleID+": "+v.Message) + } + return out +} + +func pluggable(id, name string, children ...*ast.WidgetV3) *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "pluggablewidget", + Name: name, + Properties: map[string]any{"WidgetType": id}, + Children: children, + } +} + +// An unknown widget id reaches `exec` and fails there, while `check` passes. +// The parser cannot catch it — the id is a string literal — so the validator is +// the only thing that could, and today it says nothing. +func TestValidateWidgetKind_UnknownWidgetIDIsReported(t *testing.T) { + // Needs a project: with none, the rule cannot tell an unknown widget from + // one installed in a project it cannot see — see the no-project control. + got := widgetKindViolations(t, "../../testdata/expr-checker/minimal.mpr", []*ast.WidgetV3{ + pluggable("com.acme.widget.NotAWidget", "w1"), + }) + if !containsRule(got, "MDL-WIDGET25") { + t.Fatalf("unknown widget id not reported; got %v", got) + } + if !containsText(got, "com.acme.widget.NotAWidget") { + t.Errorf("the message does not name the widget: %v", got) + } +} + +// The control for the above: a widget mxcli knows must stay silent, or the rule +// is simply "always complain". +func TestValidateWidgetKind_KnownWidgetIDIsSilent(t *testing.T) { + got := widgetKindViolations(t, "", []*ast.WidgetV3{ + pluggable("com.mendix.widget.web.combobox.Combobox", "w1"), + }) + if containsRule(got, "MDL-WIDGET25") { + t.Errorf("a known widget was reported as unknown: %v", got) + } +} + +// A container keyword the parent does not declare — `group` on a widget whose +// definition has no such object list. It parses (GROUP is in the grammar) and +// the validator used to SKIP it, because isUniversalObjectListKeyword treats +// the keyword as always-an-item wherever it appears. +// +// Tested against a synthetic definition rather than through the tree walk: no +// EMBEDDED widget declares an object list, and the fixture project has no +// extracted defs, so neither route can express the control below. +func TestValidateWidgetKind_ContainerNotDeclaredByTheParentIsReported(t *testing.T) { + registry := LoadWidgetRegistry("") + parent := &WidgetDefinition{ + MDLName: "PARENTWIDGET", + ObjectLists: []ObjectListMapping{ + {MDLContainer: "SERIES", PropertyKey: "series"}, + }, + } + got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, parent, objectListMappingSet(parent), "page X") + + if len(got) == 0 || got[0].RuleID != "MDL-WIDGET26" { + t.Fatalf("an undeclared container was not reported; got %v", got) + } + if !strings.Contains(got[0].Message, "group") { + t.Errorf("the message does not name the container: %s", got[0].Message) + } + // It must say what the parent DOES declare, or the reader is left guessing. + if !strings.Contains(got[0].Message, "series") { + t.Errorf("the message does not name the parent's real containers: %s", got[0].Message) + } +} + +// The control: the same keyword on a parent that DOES declare it stays silent. +// Without this, the rule above passes against a build that rejects everything. +func TestValidateWidgetKind_ContainerDeclaredByTheParentIsSilent(t *testing.T) { + registry := LoadWidgetRegistry("") + parent := &WidgetDefinition{ + MDLName: "PARENTWIDGET", + ObjectLists: []ObjectListMapping{ + {MDLContainer: "GROUP", PropertyKey: "groups"}, + }, + } + if got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, parent, objectListMappingSet(parent), "page X"); len(got) != 0 { + t.Errorf("the parent declares `group`, so it must not be reported: %v", got) + } +} + +// A declared CHILD SLOT is equally legitimate. Object lists are not the whole +// vocabulary of a widget body. +func TestValidateWidgetKind_DeclaredChildSlotIsSilent(t *testing.T) { + registry := LoadWidgetRegistry("") + parent := &WidgetDefinition{ + MDLName: "PARENTWIDGET", + ChildSlots: []ChildSlotMapping{{MDLContainer: "GROUP", PropertyKey: "groups"}}, + } + if got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, parent, nil, "page X"); len(got) != 0 { + t.Errorf("a declared child slot must not be reported: %v", got) + } +} + +// An unresolvable parent must silence the rule. In a project that never ran +// `widget init` the registry knows only the embedded widgets, so every real +// parent looks container-less — reporting there would bury correct MDL. +func TestValidateWidgetKind_UnresolvableParentSilencesTheContainerRule(t *testing.T) { + registry := LoadWidgetRegistry("") + if got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, nil, nil, "page X"); len(got) != 0 { + t.Errorf("with no resolvable parent the rule must stay silent: %v", got) + } +} + +// An ordinary widget nested inside a pluggable widget's slot is legitimate and +// must not be mistaken for an undeclared container. +func TestValidateWidgetKind_OrdinaryNestedWidgetIsSilent(t *testing.T) { + const fixture = "../../testdata/expr-checker/minimal.mpr" + got := widgetKindViolations(t, fixture, []*ast.WidgetV3{ + pluggable("com.mendix.widget.web.htmlelement.HTMLElement", "h", + &ast.WidgetV3{Type: "container", Name: "c1"}), + }) + if containsRule(got, "MDL-WIDGET26") { + t.Errorf("a plain container widget was reported as an undeclared container: %v", got) + } +} + +func containsRule(msgs []string, rule string) bool { + for _, m := range msgs { + if strings.Contains(m, rule) { + return true + } + } + return false +} + +func containsText(msgs []string, text string) bool { + for _, m := range msgs { + if strings.Contains(m, text) { + return true + } + } + return false +} + +// With NO project the registry holds only the embedded widgets, so every real +// project widget would look unknown. `mxcli check` without -p is the common +// case — it is how the example corpus is checked in CI — and an earlier version +// of this rule produced 14 violations in a single example file. +// +// The guard that was there (is the .mpk installed?) could not help: with no +// project there is no widgets/ to look in. This is the control that measurement +// with -p could not provide. +func TestValidateWidgetKind_NoProjectMeansNoUnknownWidgetClaims(t *testing.T) { + got := widgetKindViolations(t, "", []*ast.WidgetV3{ + pluggable("com.mendix.widget.web.htmlelement.HTMLElement", "h"), + pluggable("com.acme.widget.NotAWidget", "w1"), + }) + if containsRule(got, "MDL-WIDGET25") { + t.Errorf("claimed a widget is unknown with no project to judge against: %v", got) + } +} diff --git a/mdl/executor/validate_widget_member_refs.go b/mdl/executor/validate_widget_member_refs.go new file mode 100644 index 0000000000..07ed8678cf --- /dev/null +++ b/mdl/executor/validate_widget_member_refs.go @@ -0,0 +1,416 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/mdl/xpathrefs" +) + +// Two member positions inside a widget that reference checking walked past. +// Both passed `check --references`, passed `exec`, and failed the build. +// +// - An XPATH CONSTRAINT naming a member the constrained entity does not have +// (mendixlabs/mxcli#1049). The check resolved the entity in +// `database from Bench.Order` and never looked inside the `where […]`: +// +// where [Bench.Order_Status = 'Open'] +// mx check -> [CE1613] "The selected association 'Bench.Order_Status' +// no longer exists." +// +// - A CONTENTPARAMS value written as a `$variable/` PATH where the position +// takes an attribute reachable from the widget's context object +// (mendixlabs/mxcli#1046). The writer put the whole string in the attribute +// name, so the model came out naming an attribute that could never exist: +// +// ContentParams: [{1} = $Customer/Name] +// mx check -> [CE1613] "The selected attribute +// 'Bench.Customer.$Customer/Name' no longer exists." +// +// They sit in one file because they are the same defect in two spellings, but +// they run in different TIERS, and that difference is worth keeping: the XPath +// one is a question about the model and needs -p, while the ContentParams one is +// answerable from the statement alone. Putting the second under --references +// would mean `mxcli check page.mdl` stayed silent on a mistake it can see. + +// --------------------------------------------------------------------------- +// ContentParams: a path where a bare attribute belongs (no project needed) +// --------------------------------------------------------------------------- + +// ValidateWidgetParamPaths flags a template parameter whose value carries a +// `$variable/` prefix. +// +// An association PATH is legal here — `{1} = Customer/Name` is an attribute +// reached over an association, which the writer stores as AttributeRef plus +// steps. What is not legal is rooting that path in a variable: the parameter is +// evaluated against the widget's own context object, so there is nowhere for a +// `$Customer/` to be resolved, and the writer keeps it as part of the name. +func ValidateWidgetParamPaths(prog *ast.Program) []linter.Violation { + if prog == nil { + return nil + } + var out []linter.Violation + for _, stmt := range prog.Statements { + forEachWidget(stmt, func(w *ast.WidgetV3, where string) { + for _, prop := range []string{"ContentParams", "CaptionParams"} { + params, ok := w.Properties[prop].([]ast.ParamAssignmentV3) + if !ok { + continue + } + for _, p := range params { + text := paramValueText(p.Value) + rest, bad := templateParamDefect(text) + if !bad { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL-WIDGET24", + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s: %s parameter {%d} is written as %q — a template parameter is "+ + "evaluated against the widget's own context object, so it takes the "+ + "attribute name (or an association path to one), not a path rooted in "+ + "a variable. Write %q. Left as-is the prefix becomes part of the "+ + "attribute name and mxbuild reports CE1613 \"The selected attribute "+ + "'….%s' no longer exists\"", + where, prop, p.Index, text, rest, text), + }) + } + } + }) + } + return out +} + +// templateParamDefect decides whether a parameter value is one the writer +// cannot resolve, and what to write instead. +// +// The rule is narrower than "a variable root is wrong", because the writer +// already strips one specific prefix. Measured on 11.13.0, five shapes, each +// executed and built: +// +// OrderNo clean +// Bench.Order_Customer/Name clean (association hop) +// $currentObject/Bench.Order_Customer/Name clean (prefix stripped) +// $currentObject/OrderNo CE1613 +// $Customer/Name CE1613 +// +// So `$currentObject/` is only stripped on the association branch +// (resolveAssociationAttributePath), and falls through with the prefix intact +// when the remainder is a bare attribute. Flagging every `$` root would report +// the third line, which builds — a false error on a working page. +// +// The fifth shape in that measurement, `Assoc/Entity/Attr` (the XPath spelling), +// also fails — but deciding it needs the model to say which segments are +// associations, so it belongs in the --references tier and is not claimed here. +func templateParamDefect(text string) (fix string, bad bool) { + if !strings.HasPrefix(text, "$") { + return "", false + } + slash := strings.Index(text, "/") + if slash < 0 || slash == len(text)-1 { + return "", false + } + variable, rest := text[1:slash], text[slash+1:] + if variable == "currentObject" { + // Stripped by the writer when what follows is an association path; + // broken when it is a bare attribute. + if strings.Contains(rest, "/") { + return "", false + } + return rest, true + } + // Any other variable root is kept verbatim, whatever follows it. + return rest, true +} + +// paramValueText renders a parameter value back to the text the author wrote, +// as far as the AST preserves it. A value this cannot render is reported as +// empty and skipped, never guessed at. +func paramValueText(v any) string { + switch e := v.(type) { + case nil: + return "" + case string: + return e + case *ast.IdentifierExpr: + return e.Name + case *ast.VariableExpr: + return "$" + e.Name + case *ast.AttributePathExpr: + // The only shape that can carry the defect: a variable root plus a + // path. Rendered here rather than through expressionToString so the + // separator the author wrote is preserved. + if e.Variable == "" || len(e.Path) == 0 { + return "" + } + return "$" + e.Variable + "/" + strings.Join(e.Path, "/") + case ast.Expression: + return expressionToString(e) + } + return "" +} + +// --------------------------------------------------------------------------- +// XPath constraints: members of the entity being constrained (needs -p) +// --------------------------------------------------------------------------- + +// validateXPathMembers resolves every step of a widget's XPath constraint +// against the entity the constraint filters. +// +// Only a DATABASE source is checked, because only there is the entity named in +// the statement. An association or microflow source carries its entity +// elsewhere, and an entity this cannot establish leaves the constraint +// unchecked rather than wrongly checked — the same rule as the member resolver +// in validate_member_refs.go, and for the same reason. +func validateXPathMembers(ctx *ExecContext, prog *ast.Program) []error { + if prog == nil || !ctx.Connected() { + return nil + } + m := &execXPathModel{ctx: ctx} + var errs []error + for _, stmt := range prog.Statements { + forEachWidget(stmt, func(w *ast.WidgetV3, where string) { + ds := w.GetDataSource() + if ds == nil || ds.Where == "" || !strings.EqualFold(ds.Type, "database") { + return + } + entityQN := ds.Reference + if entityQN == "" || !strings.Contains(entityQN, ".") { + return + } + for _, bad := range unresolvableXPathSteps(ctx, m, ds.Where, entityQN) { + errs = append(errs, mdlerrors.NewValidation(fmt.Sprintf( + "%s: the constraint on %s names %q, which is neither an attribute nor an "+ + "association of it — mxbuild reports this as CE1613 \"The selected %s "+ + "'%s' no longer exists\"", + where, entityQN, bad.name, bad.kind, bad.name))) + } + }) + } + return errs +} + +// badStep is one step of a constraint that resolved to nothing. +type badStep struct { + name string + kind string // "association" or "attribute" — which mxbuild will call it +} + +// unresolvableXPathSteps walks the constraint's predicate groups and returns the +// steps that name nothing on the entity. +// +// The parse is deliberately lenient (ANTLR with the error listeners removed), so +// a group it cannot read is SKIPPED rather than reported: a tree that quietly +// omits part of its input would otherwise produce an error about text nobody +// wrote. That is the same trap xpathrefs documents for the rename path, where +// the consequence is a corrupted constraint; here it is only a false error, but +// a false error still blocks a script that builds. +func unresolvableXPathSteps(ctx *ExecContext, m xpathrefs.Model, constraint, entityQN string) []badStep { + groups := visitor.SplitXPathPredicateGroups(constraint) + if len(groups) == 0 { + groups = []string{constraint} + } + var out []badStep + for _, g := range groups { + expr, ok := visitor.ParseXPathConstraint(g) + if !ok || expr == nil { + continue + } + v := &xpathMemberVisitor{ctx: ctx, model: m} + v.walk(expr, entityQN) + out = append(out, v.bad...) + } + return out +} + +// xpathMemberVisitor walks a parsed constraint carrying the entity each step is +// evaluated against, the same traversal xpathrefs' walker performs for renames. +type xpathMemberVisitor struct { + ctx *ExecContext + model xpathrefs.Model + bad []badStep + seen map[string]bool +} + +func (v *xpathMemberVisitor) add(s badStep) { + if v.seen == nil { + v.seen = map[string]bool{} + } + if v.seen[s.name] { + return + } + v.seen[s.name] = true + v.bad = append(v.bad, s) +} + +func (v *xpathMemberVisitor) walk(expr ast.Expression, cur string) { + switch e := expr.(type) { + case nil: + return + case *ast.XPathPathExpr: + v.walkPath(e.Steps, cur) + case *ast.BinaryExpr: + v.walk(e.Left, cur) + v.walk(e.Right, cur) + case *ast.UnaryExpr: + v.walk(e.Operand, cur) + case *ast.ParenExpr: + v.walk(e.Inner, cur) + case *ast.FunctionCallExpr: + for _, a := range e.Arguments { + v.walk(a, cur) + } + case *ast.IfThenElseExpr: + v.walk(e.Condition, cur) + v.walk(e.ThenExpr, cur) + v.walk(e.ElseExpr, cur) + case *ast.SourceExpr: + v.walk(e.Expression, cur) + case *ast.IdentifierExpr: + v.noteBare(e.Name, cur) + case *ast.QualifiedNameExpr: + // A qualified name standing alone in a predicate is the shape #1049 + // reported: `[Bench.Order_Status = 'Open']`, where the name looks like + // an association and is not one. + v.noteQualified(e.QualifiedName.String(), cur) + } +} + +// noteBare checks a bare step, which Mendix evaluates as an attribute of cur. +func (v *xpathMemberVisitor) noteBare(name, cur string) { + if cur == "" || name == "" { + return + } + if resolveMemberOnEntity(v.ctx, cur, name) == memberMissing { + v.add(badStep{name: name, kind: "attribute"}) + } +} + +// noteQualified checks a `Module.Name` step, which is an association hop or an +// entity cast. Anything else names nothing. +func (v *xpathMemberVisitor) noteQualified(qn, cur string) { + if qn == "" { + return + } + if _, ok := v.model.AssociationTarget(qn, cur); ok { + return + } + if v.model.IsEntity(qn) { + return + } + // Only report when the entity it would have hung off is itself KNOWN. + // + // Without this the check fires on a constraint whose base entity the project + // does not have — a page written against a module the script creates, or run + // against the wrong app — and reports a perfectly good association as + // missing. Caught by a control: `[Bench.Order_Customer/…]` on a project with + // no Bench module was reported, and the association was real. + // + // This is the same three-valued discipline the bare-member path gets from + // resolveMemberOnEntity: could-not-establish is silence, not a finding. + if cur == "" || !v.model.IsEntity(cur) { + return + } + v.add(badStep{name: qn, kind: "association"}) +} + +func (v *xpathMemberVisitor) walkPath(steps []ast.XPathStep, cur string) { + for i, st := range steps { + next := "" + switch e := st.Expr.(type) { + case *ast.IdentifierExpr: + if i == len(steps)-1 { + v.noteBare(e.Name, cur) + } + case *ast.QualifiedNameExpr: + qn := e.QualifiedName.String() + if t, ok := v.model.AssociationTarget(qn, cur); ok { + next = t + } else if v.model.IsEntity(qn) { + next = qn + } else { + v.noteQualified(qn, cur) + } + } + if st.Predicate != nil { + v.walk(st.Predicate, next) + } + cur = next + } +} + +// execXPathModel answers xpathrefs.Model from the connected project. +type execXPathModel struct{ ctx *ExecContext } + +func (m *execXPathModel) IsEntity(qn string) bool { + b, ok := m.ctx.Backend.(entityLookupBackend) + if !ok { + return false + } + _, found := findEntityByQN(b, qn) + return found +} + +func (m *execXPathModel) AssociationTarget(qn, from string) (string, bool) { + return associationTargetFrom(m.ctx, qn, from) +} + +// --------------------------------------------------------------------------- +// Shared widget walk +// --------------------------------------------------------------------------- + +// forEachWidget visits every widget a statement carries, with a label naming +// where it is. Every widget-bearing field has to be walked: a widget the walk +// misses is a widget nothing checks, and the escape is silent both ways. +func forEachWidget(stmt ast.Statement, fn func(w *ast.WidgetV3, where string)) { + var doc string + var roots []*ast.WidgetV3 + switch s := stmt.(type) { + case *ast.CreatePageStmtV3: + doc = "page " + s.Name.String() + roots = append(roots, s.Widgets...) + for _, ph := range s.Placeholders { + if ph != nil { + roots = append(roots, ph.Widgets...) + } + } + case *ast.CreateSnippetStmtV3: + doc = "snippet " + s.Name.String() + roots = append(roots, s.Widgets...) + case *ast.CreateLayoutStmt: + doc = "layout " + s.Name.String() + roots = append(roots, s.Widgets...) + case *ast.AlterPageStmt: + doc = strings.ToLower(s.ContainerType) + " " + s.PageName.String() + for _, op := range s.Operations { + switch o := op.(type) { + case *ast.InsertWidgetOp: + roots = append(roots, o.Widgets...) + case *ast.ReplaceWidgetOp: + roots = append(roots, o.NewWidgets...) + } + } + default: + return + } + var walk func(w *ast.WidgetV3) + walk = func(w *ast.WidgetV3) { + if w == nil { + return + } + fn(w, fmt.Sprintf("%s: %s %q", doc, strings.ToLower(w.Type), w.Name)) + for _, c := range w.Children { + walk(c) + } + } + for _, w := range roots { + walk(w) + } +} diff --git a/mdl/executor/validate_widget_member_refs_test.go b/mdl/executor/validate_widget_member_refs_test.go new file mode 100644 index 0000000000..c4b82d7228 --- /dev/null +++ b/mdl/executor/validate_widget_member_refs_test.go @@ -0,0 +1,256 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// Two member positions inside a widget that reference checking walked past: +// an XPath constraint's members (mendixlabs/mxcli#1049) and a template +// parameter written as a `$variable/` path (mendixlabs/mxcli#1046). Both passed +// check, passed exec, and failed the build with CE1613. + +func paramWidget(name string, prop string, value any) *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "dynamictext", Name: name, + Properties: map[string]any{ + prop: []ast.ParamAssignmentV3{{Index: 1, Value: value}}, + }, + } +} + +func pageWith(widgets ...*ast.WidgetV3) *ast.Program { + return &ast.Program{Statements: []ast.Statement{ + &ast.CreatePageStmtV3{ + Name: ast.QualifiedName{Module: "Shop", Name: "P"}, + Widgets: widgets, + }, + }} +} + +func varPath(v string, path ...string) *ast.AttributePathExpr { + return &ast.AttributePathExpr{Variable: v, Path: path} +} + +// --------------------------------------------------------------------------- +// #1046 — ContentParams +// --------------------------------------------------------------------------- + +// A template parameter is evaluated against the widget's own context object. +// The writer strips ONE prefix and only on one branch, so the rule is narrower +// than "a variable root is wrong". Measured on 11.13.0, each shape executed and +// built: +// +// OrderNo clean +// Bench.Order_Customer/Name clean (association hop) +// $currentObject/Bench.Order_Customer/Name clean (prefix stripped) +// $currentObject/OrderNo CE1613 +// $Customer/Name CE1613 +func TestValidateWidgetParamPaths_ReportsAVariableRootedPath(t *testing.T) { + for _, tc := range []struct { + name string + variable string + path []string + want string + }{ + {"a page parameter with a bare attribute", "Customer", []string{"Name"}, "Name"}, + {"$currentObject with a bare attribute", "currentObject", []string{"OrderNo"}, "OrderNo"}, + // A non-currentObject root is kept verbatim whatever follows it, so an + // association path under one is broken too. + {"a page parameter with an association path", "Customer", + []string{"Shop.Order_Customer", "Name"}, "Shop.Order_Customer/Name"}, + } { + t.Run(tc.name, func(t *testing.T) { + v := ValidateWidgetParamPaths(pageWith( + paramWidget("dt", "ContentParams", varPath(tc.variable, tc.path...)))) + if len(v) != 1 { + t.Fatalf("got %d violations, want 1: %v", len(v), v) + } + if v[0].RuleID != "MDL-WIDGET24" { + t.Errorf("rule = %s, want MDL-WIDGET24", v[0].RuleID) + } + // The message has to say what to write instead — the fix is one + // token, and an error that only says "wrong" makes the author guess. + if !strings.Contains(v[0].Message, `Write "`+tc.want+`"`) { + t.Errorf("message should name the replacement %q: %s", tc.want, v[0].Message) + } + if !strings.Contains(v[0].Message, "CE1613") { + t.Errorf("message should name the build error it prevents: %s", v[0].Message) + } + }) + } +} + +// THE CONTROL for the half of the rule that is easy to get wrong. The writer +// DOES strip `$currentObject/` when an association path follows, so flagging +// every variable root reports a page that builds cleanly. Measured, not assumed: +// `$currentObject/Bench.Order_Customer/Name` executed and built at 0 errors. +func TestValidateWidgetParamPaths_AcceptsCurrentObjectOnAnAssociationPath(t *testing.T) { + v := ValidateWidgetParamPaths(pageWith(paramWidget("dt", "ContentParams", + varPath("currentObject", "Shop.Order_Customer", "Name")))) + if len(v) != 0 { + t.Errorf("reported a shape the writer resolves: %v", v) + } +} + +// CaptionParams is the same position under another name, and a check that +// covered only one of them would leave the other silent. +func TestValidateWidgetParamPaths_CoversCaptionParams(t *testing.T) { + v := ValidateWidgetParamPaths(pageWith( + paramWidget("btn", "CaptionParams", varPath("Order", "OrderNo")))) + if len(v) != 1 { + t.Fatalf("got %d violations, want 1: %v", len(v), v) + } +} + +// CONTROL: the spellings that are correct must stay silent. An association PATH +// is legal here — it is an attribute reached over an association, which the +// writer stores as AttributeRef plus steps — so only the VARIABLE root is wrong. +func TestValidateWidgetParamPaths_AcceptsWhatIsLegal(t *testing.T) { + for _, tc := range []struct { + name string + value any + }{ + {"a bare attribute", &ast.IdentifierExpr{Name: "Name"}}, + {"an association hop then an attribute", &ast.IdentifierExpr{Name: "Shop.Order_Customer/Name"}}, + {"a bare variable with no path", &ast.VariableExpr{Name: "Customer"}}, + {"nothing at all", nil}, + } { + t.Run(tc.name, func(t *testing.T) { + if v := ValidateWidgetParamPaths(pageWith( + paramWidget("dt", "ContentParams", tc.value))); len(v) != 0 { + t.Errorf("reported a legal parameter: %v", v) + } + }) + } +} + +// It needs no project, which is the point of putting it in the unconditional +// pass: `mxcli check page.mdl` would otherwise stay silent on a mistake it can +// see from the statement alone. +func TestValidateWidgetParamPaths_NeedsNoProject(t *testing.T) { + v := ValidateProgram(pageWith( + paramWidget("dt", "ContentParams", varPath("Customer", "Name"))), "") + var found bool + for _, x := range v { + if x.RuleID == "MDL-WIDGET24" { + found = true + } + } + if !found { + t.Errorf("MDL-WIDGET24 did not fire in the no-project pass: %v", v) + } +} + +// --------------------------------------------------------------------------- +// #1049 — XPath constraint members +// --------------------------------------------------------------------------- + +func dbGrid(entityQN, where string) *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "datagrid", Name: "g", + Properties: map[string]any{ + "DataSource": &ast.DataSourceV3{ + Type: "database", Reference: entityQN, Where: where, + }, + }, + } +} + +func TestValidateXPathMembers_ReportsAMemberThatIsNeither(t *testing.T) { + ctx := memberFixture(t) + errs := validateXPathMembers(ctx, pageWith( + dbGrid("Shop.Order", "[Shop.Order_Status = 'Open']"))) + + if len(errs) != 1 { + t.Fatalf("got %d errors, want 1: %v", len(errs), errs) + } + for _, want := range []string{"Shop.Order_Status", "Shop.Order", "CE1613"} { + if !strings.Contains(errs[0].Error(), want) { + t.Errorf("error should mention %q: %v", want, errs[0]) + } + } +} + +func TestValidateXPathMembers_ReportsABareAttributeThatIsMissing(t *testing.T) { + ctx := memberFixture(t) + errs := validateXPathMembers(ctx, pageWith( + dbGrid("Shop.Order", "[Nonsense = 'x']"))) + if len(errs) != 1 || !strings.Contains(errs[0].Error(), "Nonsense") { + t.Fatalf("a bare attribute that does not exist went unreported: %v", errs) + } +} + +// CONTROL: every constraint that resolves must stay silent. Each of these is a +// shape a real page uses. +func TestValidateXPathMembers_AcceptsWhatResolves(t *testing.T) { + ctx := memberFixture(t) + for _, tc := range []struct{ name, entity, where string }{ + {"an attribute the entity declares", "Shop.Order", "[Status = 'Open']"}, + {"an attribute INHERITED from the generalization", "Shop.Order", "[Code = 'x']"}, + {"an association hop, then the target's attribute", + "Shop.Order", "[Shop.Order_Customer/Shop.Customer/Name = 'x']"}, + {"no constraint at all", "Shop.Order", ""}, + } { + t.Run(tc.name, func(t *testing.T) { + if errs := validateXPathMembers(ctx, pageWith( + dbGrid(tc.entity, tc.where))); len(errs) != 0 { + t.Errorf("reported a constraint that resolves: %v", errs) + } + }) + } +} + +// THE CONTROL that a real false positive produced. An entity the project does +// not have makes the whole constraint unanswerable — reporting from it called a +// perfectly good association missing, on a page written against a module the +// script creates or run against the wrong app. +func TestValidateXPathMembers_SilentWhenTheBaseEntityIsUnknown(t *testing.T) { + ctx := memberFixture(t) + for _, where := range []string{ + "[Shop.Order_Customer/Shop.Customer/Name = 'x']", // a REAL association + "[Anything = 'x']", + "[Made.Up_Thing = 'x']", + } { + if errs := validateXPathMembers(ctx, pageWith( + dbGrid("Nowhere.Missing", where))); len(errs) != 0 { + t.Errorf("reported against an entity the project does not have: %v", errs) + } + } +} + +// Only a DATABASE source names its entity in the statement. Anything else +// carries it elsewhere, so the constraint is left unchecked rather than checked +// against the wrong entity. +func TestValidateXPathMembers_OnlyChecksADatabaseSource(t *testing.T) { + ctx := memberFixture(t) + w := &ast.WidgetV3{ + Type: "datagrid", Name: "g", + Properties: map[string]any{ + "DataSource": &ast.DataSourceV3{ + Type: "association", Reference: "Shop.Order", Where: "[Nonsense = 'x']", + }, + }, + } + if errs := validateXPathMembers(ctx, pageWith(w)); len(errs) != 0 { + t.Errorf("checked a non-database source: %v", errs) + } +} + +// A constraint mxcli cannot parse is skipped, not reported. The XPath parse runs +// with ANTLR's error listeners removed and can hand back a tree that quietly +// omits part of its input, so reporting from a partial tree means an error about +// text nobody wrote. +func TestValidateXPathMembers_SkipsAnUnparseableConstraint(t *testing.T) { + ctx := memberFixture(t) + for _, where := range []string{"[", "[[[", "[ = = ]"} { + if errs := validateXPathMembers(ctx, pageWith( + dbGrid("Shop.Order", where))); len(errs) != 0 { + t.Errorf("reported from a constraint it could not read (%q): %v", where, errs) + } + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 7ee78a8ae0..954d7f76cb 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -62,6 +62,11 @@ func LoadWidgetRegistry(projectPath string) *WidgetRegistry { if projectPath != "" { _ = registry.LoadUserDefinitions(projectPath) registry.projectPath = projectPath + // The validator and DESCRIBE WIDGET must agree about which properties a + // widget has; they read different sources, so the definition is topped up + // from the same .mpk DESCRIBE parses. See + // widget_known_props_from_mpk.go for why this is not a list of nine. + enrichKnownPropertiesFromMPK(registry, projectPath) } return registry } @@ -116,6 +121,9 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc } mapping := parentObjectLists[strings.ToUpper(w.Type)] isObjectListItem := mapping != nil || isUniversalObjectListKeyword(w.Type) + // Slice 0: is this a widget at all, and does the parent declare this + // container? Both were previously left to `exec`. + out = append(out, validateWidgetKind(w, registry, lookupWidgetDef(parent, registry), parentObjectLists, locationPrefix)...) out = append(out, validatePluggableWidgetProperties(w, registry, locationPrefix)...) // #928: contentparams with no `{N}` placeholder to consume them. if lookupWidgetDef(w, registry) != nil { @@ -135,7 +143,14 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // widgets get the stricter def.json check (MDL-WIDGET01) above, and // object-list items are validated by the object-list engine. def := lookupWidgetDef(w, registry) - if def == nil && !isObjectListItem { + // A generic widget type that resolved to nothing is already reported as + // MDL-WIDGET25 (the kind is wrong). Validating its properties on top of + // that says the kind is fine and the property is not, which points at + // the wrong token — measured on `htmlelemnt frame (tagName: 'div')`, + // which drew a `tagName` warning beside the real error. A built-in + // (TypeIsGeneric false) keeps the check, since its properties are the + // only thing that can be wrong about it. + if def == nil && !isObjectListItem && !w.TypeIsGeneric { out = append(out, validateStaticWidgetUnknownProps(w, locationPrefix)...) // #928: `editable:` on a widget Mendix gives no editability — same // "silently dropped on write" family, but the flat property @@ -642,7 +657,7 @@ func isKnownStaticWidgetProp(key string) bool { // separately (MDL-WIDGET01) and must not reach here. func validateStaticWidgetUnknownProps(w *ast.WidgetV3, locationPrefix string) []linter.Violation { var out []linter.Violation - for key := range w.Properties { + for _, key := range sortedPropertyKeys(w) { if isKnownStaticWidgetProp(key) { continue } @@ -695,7 +710,7 @@ func validateDynamicTextFormatting(w *ast.WidgetV3, locationPrefix string) []lin // (1) Format keys placed at the widget level are silently dropped on write — // formatting is per-parameter. Flag them with the correct location. if strings.EqualFold(w.Type, "dynamictext") { - for key := range w.Properties { + for _, key := range sortedPropertyKeys(w) { if paramFormatKeys[strings.ToLower(key)] { out = append(out, linter.Violation{ RuleID: "MDL-WIDGET18", @@ -1044,7 +1059,7 @@ func validatePluggableWidgetProperties(w *ast.WidgetV3, registry *WidgetRegistry knownUnmapped := knownUnmappedProperties(def, allowed) var out []linter.Violation - for key := range w.Properties { + for _, key := range sortedPropertyKeys(w) { // Builtin property names (Label, Class, Visible, DataSource, …) are // MDL-recognized keywords that the widget engine routes via a // dedicated path rather than via propertyMappings. Accept them @@ -1153,6 +1168,16 @@ func addMappingNames(add func(string), m PropertyMapping) { add(m.PropertyKey) } add(m.Source) + // The aliases are the names people are TOLD to write, so they have to be + // accepted here — the builder already resolves them (widget_engine.go), and + // the knownProperties set in widget_defs.go already walks them. Leaving them + // out made the validator the odd one out of three readers of the same + // def.json: `ValueAttribute: Total` on a PieChart persisted correctly and + // was still reported as MDL-WIDGET01 "has no property", which — because exec + // refuses a script with errors — blocked the page from being written at all. + for _, a := range m.MdlAliases { + add(a) + } } // readsFixedASTSlot reports whether an operation's value is resolved from a @@ -1525,3 +1550,33 @@ func mappingOperationFor(def *WidgetDefinition, propertyKey string) string { } return "" } + +// sortedPropertyKeys returns a widget's property keys in a stable order. +// +// A validator that appends one violation per property key was iterating the map +// directly, so `mxcli check` printed the same warnings in a different order from +// one run to the next. Measured before the fix: two runs of the same binary over +// mdl-examples/ disagreed on 11 of 515 scripts. +// +// Nothing was wrong with the diagnostics — but "the output is stable" is what +// makes a before/after diff of `check` usable as a measurement, and it was not. +// This surfaced while diffing check output across the corpus to size the +// grammar change for slices 2-3: the noise floor of the tool was larger than +// the signal being looked for. +// +// The three call sites are the ones that emit PER KEY (MDL-WIDGET07, WIDGET17, +// WIDGET18). Two other loops over w.Properties do a case-insensitive LOOKUP and +// break on the first hit; those are left alone, since they are only +// order-sensitive when a widget carries two keys differing solely in case, and +// picking either is equally correct. +func sortedPropertyKeys(w *ast.WidgetV3) []string { + if w == nil { + return nil + } + out := make([]string, 0, len(w.Properties)) + for k := range w.Properties { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/mdl/executor/validate_widgets_order_test.go b/mdl/executor/validate_widgets_order_test.go new file mode 100644 index 0000000000..117161d134 --- /dev/null +++ b/mdl/executor/validate_widgets_order_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// widgetWithUnknownProps builds a static widget carrying several properties no +// builder consumes, so every one produces an MDL-WIDGET07 warning. Eight keys +// make an accidental pass vanishingly unlikely: with unsorted map iteration the +// chance of Go handing back the same order twice is 1/8!. +func widgetWithUnknownProps() *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "container", + Name: "c1", + Properties: map[string]any{ + "zeta": "1", "alpha": "2", "mike": "3", "delta": "4", + "omega": "5", "bravo": "6", "kilo": "7", "sierra": "8", + }, + } +} + +func messagesOf(vs []linter.Violation) []string { + out := make([]string, 0, len(vs)) + for _, v := range vs { + out = append(out, v.Message) + } + return out +} + +// `mxcli check` printed the same warnings in a different order from one run to +// the next, because the validator iterated w.Properties directly. Nothing was +// wrong with the diagnostics, but it made a before/after diff of check output +// useless as a measurement — two runs of the same binary over mdl-examples/ +// disagreed on 11 of 515 scripts, which was more noise than the grammar change +// being measured produced signal. +func TestStaticWidgetUnknownProps_OrderIsStable(t *testing.T) { + first := messagesOf(validateStaticWidgetUnknownProps(widgetWithUnknownProps(), "page M.P")) + if len(first) != 8 { + t.Fatalf("got %d violations, want 8 — the fixture must produce one per unknown key, or this test proves nothing", len(first)) + } + + for i := 0; i < 50; i++ { + got := messagesOf(validateStaticWidgetUnknownProps(widgetWithUnknownProps(), "page M.P")) + if len(got) != len(first) { + t.Fatalf("run %d produced %d violations, want %d", i, len(got), len(first)) + } + for j := range first { + if got[j] != first[j] { + t.Fatalf("run %d differs at position %d:\n got %q\nwant %q\nwidget property warnings must not depend on map iteration order", + i, j, got[j], first[j]) + } + } + } +} + +// Stable is necessary but not sufficient — it must also be a stable order a +// reader can predict. Sorted by property key is the one the fix chose. +func TestStaticWidgetUnknownProps_OrderIsSorted(t *testing.T) { + w := widgetWithUnknownProps() + got := messagesOf(validateStaticWidgetUnknownProps(w, "page M.P")) + + keys := make([]string, 0, len(w.Properties)) + for k := range w.Properties { + keys = append(keys, k) + } + sort.Strings(keys) + + if len(got) != len(keys) { + t.Fatalf("got %d violations for %d properties", len(got), len(keys)) + } + for i, k := range keys { + if !strings.Contains(got[i], "`"+k+"`") { + t.Errorf("violation %d = %q, want it to be about property %q — warnings should follow sorted key order", i, got[i], k) + } + } +} + +// sortedPropertyKeys is the shared helper; a nil widget must not panic, since +// the validators are called on trees built from partial parses. +func TestSortedPropertyKeys(t *testing.T) { + if got := sortedPropertyKeys(nil); got != nil { + t.Errorf("sortedPropertyKeys(nil) = %v, want nil", got) + } + if got := sortedPropertyKeys(&ast.WidgetV3{}); len(got) != 0 { + t.Errorf("sortedPropertyKeys(no properties) = %v, want empty", got) + } + w := &ast.WidgetV3{Properties: map[string]any{"b": 1, "a": 2, "c": 3}} + got := sortedPropertyKeys(w) + want := []string{"a", "b", "c"} + for i := range want { + if got[i] != want[i] { + t.Errorf("sortedPropertyKeys = %v, want %v", got, want) + break + } + } +} diff --git a/mdl/executor/widget_defs.go b/mdl/executor/widget_defs.go index 7f838d14be..7882892b36 100644 --- a/mdl/executor/widget_defs.go +++ b/mdl/executor/widget_defs.go @@ -796,12 +796,17 @@ func widgetDocMarkdown(mpkDef *mpk.WidgetDefinition, def *WidgetDefinition, mdlN buf.WriteString(fmt.Sprintf("%s '%s' widget1", prefix, mpkDef.ID)) if def != nil && (len(def.ChildSlots) > 0 || len(def.ObjectLists) > 0) { buf.WriteString(" {\n") - for _, slot := range def.ChildSlots { - buf.WriteString(fmt.Sprintf(" %s {\n -- widgets for `%s`\n }\n", strings.ToLower(slot.MDLContainer), slot.PropertyKey)) + // Names are required — `controlbar cb1 { … }`, never `controlbar { … }` + // — and must be unique within the page, so they are numbered. Emitting + // the nameless form made even the slots that DO parse unusable as + // written (mendixlabs/mxcli#1036). + for i, slot := range def.ChildSlots { + buf.WriteString(fmt.Sprintf(" %s slot%d {\n -- widgets for `%s`\n }\n", + strings.ToLower(slot.MDLContainer), i+1, slot.PropertyKey)) } - for _, ol := range def.ObjectLists { - itemKw := strings.ToLower(ol.MDLContainer) - buf.WriteString(fmt.Sprintf(" %s item1 -- one entry of `%s`\n", itemKw, ol.PropertyKey)) + for i, ol := range def.ObjectLists { + buf.WriteString(fmt.Sprintf(" %s item%d -- one entry of `%s`\n", + strings.ToLower(ol.MDLContainer), i+1, ol.PropertyKey)) } buf.WriteString("}\n") } else { diff --git a/mdl/executor/widget_describe.go b/mdl/executor/widget_describe.go new file mode 100644 index 0000000000..a61b14b839 --- /dev/null +++ b/mdl/executor/widget_describe.go @@ -0,0 +1,834 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "io" + "path/filepath" + "sort" + "strings" + + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" + mwidgets "github.com/mendixlabs/mxcli/modelsdk/widgets" + mmpk "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// DescribeWidget assembles everything mxcli knows about one widget: its +// properties (key, type, caption, category, required, default, enum options) +// and the dynamic rules its editor uses to hide properties under some +// configurations. +// +// It exists in the executor rather than in cmd/ so that the MDL statement +// `DESCRIBE WIDGET x` and the CLI `mxcli widget describe x` are the same code. +// That is the point of the statement: a widget was the one MDL extension point +// with no in-language DESCRIBE, which is why `mxcli widget init` had to generate +// documentation at all — and why that documentation could drift from what the +// parser accepts (mendixlabs/mxcli#1036). +// +// arg is an MDL keyword (COMBOBOX), a widget id +// (com.mendix.widget.web.combobox.Combobox), or one of a few built-in aliases. +// projectPath may be empty, in which case only mxcli's embedded knowledge is +// available and the answer is correspondingly thinner. +func DescribeWidget(arg, projectPath string) (*WidgetDescription, error) { + registry, err := NewWidgetRegistry() + if err != nil { + return nil, mdlerrors.NewBackend("widget registry init", err) + } + if projectPath != "" { + _ = registry.LoadUserDefinitions(projectPath) + } + + widgetID, def := resolveWidgetTarget(registry, arg) + if widgetID == "" { + return nil, widgetNotFoundError(registry, arg) + } + + desc := WidgetDescription{WidgetID: widgetID} + if def != nil { + desc.MDLName = def.MDLName + desc.Kind = def.WidgetKind + } + if desc.Kind == "" { + desc.Kind = "pluggable" + } + + // Properties + version: prefer the project's installed .mpk (version-accurate, + // and the only place a Marketplace widget appears); else mxcli's embedded + // template. + if projectPath != "" { + if dir := projectDirOf(projectPath); dir != "" { + if mpkPath, ferr := mmpk.FindMPK(dir, widgetID); ferr == nil && mpkPath != "" { + if wd, perr := mmpk.ParseMPKForWidget(mpkPath, widgetID); perr == nil && wd != nil { + desc.Name = wd.Name + desc.Version = wd.Version + desc.Source = "project .mpk" + desc.Properties = propsFromMPK(wd) + desc.Rules, desc.RuleCoverage = rulesFromProject(mpkPath, widgetID) + } + } + } + } + desc.Containers = describeContainers(def) + + if desc.Source == "" { + tmpl, terr := mwidgets.GetTemplate(widgetID) + if terr != nil || tmpl == nil { + return nil, mdlerrors.NewNotFoundMsg("widget", arg, + "no installed .mpk and no embedded template for "+arg+ + " — open the project with -p to inspect a widget it has installed") + } + desc.Name = tmpl.Name + desc.Version = tmpl.Version + desc.Source = "embedded template" + desc.Properties = propsFromTemplate(tmpl.Type) + if def != nil { + desc.Rules = rulesFromDef(def.PropertyVisibility) + } + } + desc.defDefaults = definitionDefaults(def) + desc.Example, desc.OmittedFromExample = buildUsageExample(desc) + return &desc, nil +} + +type DescribedProperty struct { + Key string `json:"key"` + Type string `json:"type"` + Caption string `json:"caption,omitempty"` + Category string `json:"category,omitempty"` + Required bool `json:"required"` + Default string `json:"default,omitempty"` + System bool `json:"system,omitempty"` + Enum []string `json:"enum,omitempty"` + Children []DescribedProperty `json:"children,omitempty"` +} + +// DescribedRule is one dynamic (visibility) rule of a widget's discovered format. +type DescribedRule struct { + Property string `json:"property"` + HiddenWhen string `json:"hiddenWhen"` + // Cond is the same condition in machine form. Kept alongside the English + // so the usage example can EVALUATE it: a widget's required properties are + // required only where visible, and Combo box lists eleven bindings of which + // its mutually exclusive options-source modes leave about two. + Cond *types.WidgetVisibilityCondition `json:"-"` + // Nested marks a rule about an object-list ITEM's property rather than the + // widget's own. Those are evaluated against the item, never the widget. + Nested bool `json:"-"` +} + +// WidgetDescription is the full inspection result (also the JSON shape). +type WidgetDescription struct { + WidgetID string `json:"widgetId"` + MDLName string `json:"mdlName,omitempty"` + Name string `json:"name,omitempty"` + Version string `json:"version,omitempty"` + Source string `json:"source"` // "project .mpk" | "embedded template" + Kind string `json:"kind,omitempty"` + Properties []DescribedProperty `json:"properties"` + Rules []DescribedRule `json:"dynamicRules"` + RuleCoverage string `json:"ruleCoverage,omitempty"` + // Containers a widget's body can hold: child slots (a curly-brace block of + // widgets) and object lists (repeating entries). Reported with whether MDL + // can currently express each one, which is NOT a given — see + // DescribedContainer.Authorable. + Containers []DescribedContainer `json:"containers,omitempty"` + // Example is re-executable MDL placing this widget, and OmittedFromExample + // says what it leaves out. Both are derived by parsing, so the example is + // guaranteed to parse and widens on its own as the grammar does. + Example string `json:"example,omitempty"` + OmittedFromExample []string `json:"omittedFromExample,omitempty"` + + // defDefaults is the value mxcli's own WidgetDefinition gives each property + // when a script does not set one — the mapping's `default`, else a primitive + // mapping's `value`. Unexported, so it never reaches the JSON output; it + // exists only so the example's hide-rule narrowing resolves a property to + // the SAME value MDL-WIDGET10 will. + // + // The .mpk alone is not enough: a selection property declares no + // defaultValue there, so gallery's `itemSelection` looked indeterminable + // and the example emitted the `keepSelection` that "Single" hides. + defDefaults map[string]string +} + +// DescribedContainer is one child slot or object list of a widget. +type DescribedContainer struct { + Keyword string `json:"keyword"` + PropertyKey string `json:"propertyKey"` + Kind string `json:"kind"` // "child slot" | "object list" + ItemKeys []string `json:"itemKeys,omitempty"` + // Authorable reports whether ` name (…)` actually parses inside a + // widget body today. It is derived by parsing a probe, never from a list: + // the bug this description exists to help with (mendixlabs/mxcli#1036) was + // two lists of keywords with nothing comparing them, and a third list here + // would be the same mistake one layer up. + Authorable bool `json:"authorable"` + + // items carries each sub-property's writable value as the WIDGET DEFINITION + // records it — the mapping's `default`/`value` and its enumValues. Unexported, + // so the JSON shape is unchanged. + // + // The .mpk is not always enough: ParseMPKForWidget returns 0 children for a + // PopupMenu's `basicItems`, while the definition carries + // {"propertyKey":"itemType","value":"item","enumValues":["item","divider"]}. + // MDL-WIDGET08 reads the definition, so the example has to as well or the two + // disagree about the same sub-property. + items []DescribedProperty +} + +func resolveWidgetTarget(registry *WidgetRegistry, arg string) (string, *WidgetDefinition) { + if strings.Contains(arg, ".") { + if def, ok := registry.GetByWidgetID(arg); ok { + return arg, def + } + return arg, nil // unknown to the registry, but a valid id to look up in the project + } + upper := strings.ToUpper(arg) + if def, ok := registry.Get(upper); ok { + return def.WidgetID, def + } + // Well-known widgets that are special-cased in the executor (no .def.json in the + // registry) but that users still name by keyword. + if id, ok := builtinWidgetAliases[upper]; ok { + def, _ := registry.GetByWidgetID(id) + return id, def + } + return "", nil +} + +// builtinWidgetAliases maps MDL keywords for executor-special-cased widgets (which +// have no .def.json registry entry) to their widget ids, so `widget describe` can +// resolve them by the same friendly names users write in MDL. +var builtinWidgetAliases = map[string]string{ + "DATAGRID": "com.mendix.widget.web.datagrid.Datagrid", + "DATAGRID2": "com.mendix.widget.web.datagrid.Datagrid", +} + +// widgetNotFoundError builds a helpful error listing the known MDL names. +func widgetNotFoundError(registry *WidgetRegistry, arg string) error { + var names []string + for _, d := range registry.All() { + if d.MDLName != "" { + names = append(names, d.MDLName) + } + } + for alias := range builtinWidgetAliases { + names = append(names, strings.ToLower(alias)) + } + sort.Strings(names) + return fmt.Errorf("unknown widget %q — use an MDL keyword (%s) or a full widget id (com.mendix.widget…). Run `mxcli widget list` to see all", + arg, strings.Join(names, ", ")) +} + +// projectDirOf returns the directory containing widgets/ for a project path +// (accepts either the .mpr file or its directory). +func projectDirOf(projectPath string) string { + if strings.EqualFold(filepath.Ext(projectPath), ".mpr") { + return filepath.Dir(projectPath) + } + return projectPath +} + +// propsFromMPK builds described properties from a parsed .mpk definition, in the +// widget's declared order (regular + system interleaved). +func propsFromMPK(wd *mmpk.WidgetDefinition) []DescribedProperty { + order := wd.AllTopLevel + if len(order) == 0 { + order = wd.Properties + } + out := make([]DescribedProperty, 0, len(order)) + for _, p := range order { + out = append(out, describedPropFromMPK(p)) + } + return out +} + +func describedPropFromMPK(p mmpk.PropertyDef) DescribedProperty { + dp := DescribedProperty{ + Key: p.Key, + Type: p.Type, + Caption: p.Caption, + Category: p.Category, + Required: p.Required, + Default: p.DefaultValue, + System: p.IsSystem, + } + if dp.System && dp.Type == "" { + dp.Type = "system" + } + for _, ev := range p.EnumValues { + dp.Enum = append(dp.Enum, ev.Key) + } + for _, c := range p.Children { + dp.Children = append(dp.Children, describedPropFromMPK(c)) + } + return dp +} + +// propsFromTemplate walks an embedded template's Type map (ObjectType.PropertyTypes) +// to build described properties. Used when no project .mpk is available. +func propsFromTemplate(typ map[string]any) []DescribedProperty { + objType, _ := typ["ObjectType"].(map[string]any) + pts, _ := objType["PropertyTypes"].([]any) + var out []DescribedProperty + for _, pt := range pts { + m, ok := pt.(map[string]any) + if !ok { + continue // leading array marker + } + out = append(out, describedPropFromTemplate(m)) + } + return out +} + +func describedPropFromTemplate(m map[string]any) DescribedProperty { + dp := DescribedProperty{ + Key: asString(m["PropertyKey"]), + Caption: asString(m["Caption"]), + Category: asString(m["Category"]), + } + vt, _ := m["ValueType"].(map[string]any) + if vt != nil { + dp.Type = asString(vt["Type"]) + dp.Default = asString(vt["DefaultValue"]) + if r, ok := vt["Required"].(bool); ok { + dp.Required = r + } + if evs, ok := vt["EnumerationValues"].([]any); ok { + for _, ev := range evs { + if em, ok := ev.(map[string]any); ok { + if k := asString(em["_Key"]); k != "" { + dp.Enum = append(dp.Enum, k) + } + } + } + } + if nested, ok := vt["ObjectType"].(map[string]any); ok { + if npts, ok := nested["PropertyTypes"].([]any); ok { + for _, npt := range npts { + if nm, ok := npt.(map[string]any); ok { + dp.Children = append(dp.Children, describedPropFromTemplate(nm)) + } + } + } + } + } + dp.System = isSystemPropKey(dp.Key) + return dp +} + +func isSystemPropKey(key string) bool { + switch key { + case "Label", "Visibility", "Editability", "Name", "TabIndex": + return true + } + return false +} + +// rulesFromProject extracts dynamic rules from the project's installed .mpk editor +// config, returning the rules and a coverage note (recognized / total hide-calls). +func rulesFromProject(mpkPath, widgetID string) ([]DescribedRule, string) { + rules, recognized, total := ExtractWidgetVisibilityStats(mpkPath, widgetID) + coverage := "" + if total > 0 { + coverage = fmt.Sprintf("%d of %d editor hide-rules recognized", recognized, total) + } + return rulesToDescribed(rules), coverage +} + +func rulesFromDef(rules []types.WidgetVisibilityRule) []DescribedRule { + return rulesToDescribed(rules) +} + +func rulesToDescribed(rules []types.WidgetVisibilityRule) []DescribedRule { + out := make([]DescribedRule, 0, len(rules)) + for _, r := range rules { + if r.HiddenWhen == nil { + continue + } + out = append(out, DescribedRule{ + Property: r.PropertyKey, + HiddenWhen: conditionText(r.HiddenWhen), + Cond: r.HiddenWhen, + Nested: r.Nested(), + }) + } + sort.Slice(out, func(i, j int) bool { return out[i].Property < out[j].Property }) + return out +} + +// conditionText renders a visibility condition as readable English. +func conditionText(c *types.WidgetVisibilityCondition) string { + switch c.Operator { + case "eq": + return fmt.Sprintf("%s = %q", c.PropertyKey, c.Value) + case "ne": + return fmt.Sprintf("%s ≠ %q", c.PropertyKey, c.Value) + case "truthy": + return fmt.Sprintf("%s is set", c.PropertyKey) + case "falsy": + return fmt.Sprintf("%s is not set", c.PropertyKey) + default: + return fmt.Sprintf("%s %s %q", c.PropertyKey, c.Operator, c.Value) + } +} + +func PrintWidgetDescription(out io.Writer, d WidgetDescription) { + title := d.Name + if title == "" { + title = d.WidgetID + } + fmt.Fprintf(out, "Widget: %s", title) + if d.MDLName != "" { + fmt.Fprintf(out, " (%s)", d.MDLName) + } + fmt.Fprintln(out) + fmt.Fprintf(out, " ID: %s\n", d.WidgetID) + if d.Version != "" { + fmt.Fprintf(out, " Version: %s\n", d.Version) + } + fmt.Fprintf(out, " Kind: %s\n", d.Kind) + fmt.Fprintf(out, " Source: %s\n", d.Source) + + fmt.Fprintf(out, "\nProperties (%d):\n", countProps(d.Properties)) + printProps(out, d.Properties, 0) + + fmt.Fprintf(out, "\nDynamic property rules (%d):\n", len(d.Rules)) + if len(d.Rules) == 0 { + fmt.Fprintln(out, " (none discovered)") + } + for _, r := range d.Rules { + fmt.Fprintf(out, " %-40s hidden when %s\n", r.Property, r.HiddenWhen) + } + if d.RuleCoverage != "" { + fmt.Fprintf(out, " — %s\n", d.RuleCoverage) + } + + if d.Example != "" { + fmt.Fprintf(out, "\nMDL example (parses as written):\n") + for _, line := range strings.Split(d.Example, "\n") { + fmt.Fprintf(out, " %s\n", line) + } + if len(d.OmittedFromExample) > 0 { + fmt.Fprintf(out, " -- omitted: %s\n", strings.Join(d.OmittedFromExample, "; ")) + } + } + + if len(d.Containers) > 0 { + fmt.Fprintf(out, "\nBody containers (%d):\n", len(d.Containers)) + for _, c := range d.Containers { + mark := " authorable" + if !c.Authorable { + mark = " NOT authorable from MDL yet" + } + fmt.Fprintf(out, " %-34s %-12s -> %s%s\n", c.Keyword, c.Kind, c.PropertyKey, mark) + if len(c.ItemKeys) > 0 { + fmt.Fprintf(out, " %-34s items: %s\n", "", strings.Join(c.ItemKeys, ", ")) + } + } + } +} + +func countProps(props []DescribedProperty) int { + n := 0 + for _, p := range props { + n++ + n += countProps(p.Children) + } + return n +} + +func printProps(out interface{ Write([]byte) (int, error) }, props []DescribedProperty, depth int) { + indent := strings.Repeat(" ", depth+1) + for _, p := range props { + req := "" + if p.Required { + req = " required" + } + sys := "" + if p.System { + sys = " [system]" + } + line := fmt.Sprintf("%s%-34s %-13s", indent, p.Key, p.Type) + extra := strings.TrimRight(req+sys, " ") + if p.Default != "" { + extra = strings.TrimSpace(extra + " default=" + p.Default) + } + if len(p.Enum) > 0 { + extra = strings.TrimSpace(extra + " {" + strings.Join(p.Enum, "|") + "}") + } + if p.Category != "" { + extra = strings.TrimSpace(extra + " (" + p.Category + ")") + } + fmt.Fprintf(out, "%s %s\n", strings.TrimRight(line, " "), extra) + if len(p.Children) > 0 { + printProps(out, p.Children, depth+1) + } + } +} + +// describeWidgetStmt is the DESCRIBE WIDGET handler. It prints exactly what +// `mxcli widget describe` prints, because it is the same function — see +// DescribeWidget for why that matters. +// +// The widget is named by MDL keyword or widget id; unlike every other DESCRIBE +// there is no qualified name, because a widget definition is not a document in +// the model. It comes from a package in the project (or from mxcli's embedded +// set), which is also why this reads no backend and works with no project open. +func describeWidgetStmt(ctx *ExecContext, name string) error { + if name == "" { + return mdlerrors.NewValidation("DESCRIBE WIDGET needs a widget: an MDL keyword (combobox) or a widget id ('com.mendix.widget.web.combobox.Combobox')") + } + projectPath := "" + if ctx != nil && ctx.Backend != nil { + projectPath = ctx.Backend.Path() + } + desc, err := DescribeWidget(name, projectPath) + if err != nil { + return err + } + PrintWidgetDescription(ctx.Output, *desc) + return nil +} + +// describeContainers lists a widget's child slots and object lists, each marked +// with whether MDL can currently express it. +func describeContainers(def *WidgetDefinition) []DescribedContainer { + if def == nil { + return nil + } + var out []DescribedContainer + for _, cs := range def.ChildSlots { + kw := strings.ToLower(cs.MDLContainer) + out = append(out, DescribedContainer{ + Keyword: kw, PropertyKey: cs.PropertyKey, Kind: "child slot", + Authorable: containerKeywordParses(kw, true), + }) + } + for _, ol := range def.ObjectLists { + kw := strings.ToLower(ol.MDLContainer) + c := DescribedContainer{ + Keyword: kw, PropertyKey: ol.PropertyKey, Kind: "object list", + Authorable: containerKeywordParses(kw, false), + } + for _, ip := range ol.ItemProperties { + c.ItemKeys = append(c.ItemKeys, ip.PropertyKey) + def := ip.Default + if def == "" && ip.Operation == "primitive" { + def = ip.Value + } + c.items = append(c.items, DescribedProperty{ + Key: ip.PropertyKey, Type: ip.Operation, Default: def, Enum: ip.EnumValues, + }) + } + out = append(out, c) + } + sort.Slice(out, func(i, j int) bool { return out[i].Keyword < out[j].Keyword }) + return out +} + +// containerKeywordParses answers "can I write this inside a widget body?" by +// parsing a minimal page and checking for errors — deriving the answer from the +// grammar itself rather than restating it. +func containerKeywordParses(keyword string, slot bool) bool { + if keyword == "" { + return false + } + body := keyword + " probe1 (x: 'y')" + if slot { + body = keyword + " probe1 { dynamictext t (Content: 'x') }" + } + src := "create page Probe.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + + " pluggablewidget 'probe.Widget' pw {\n " + body + "\n }\n}\n" + _, errs := visitor.Build(src) + return len(errs) == 0 +} + +// buildUsageExample renders MDL that places this widget, and returns it with a +// list of what it left out. +// +// Two rules make it worth printing at all, both learned from the generated .md +// this replaces (mendixlabs/mxcli#1036): +// +// 1. It emits only what PARSES. The head form, and every container, is chosen +// by probing the real parser — so the example corrects itself as the grammar +// gains ground, and cannot drift the way a hand-written template did. +// 2. It says what it omitted and why. The .md's example silently included +// containers that could not be written, which is what made it misleading +// rather than merely incomplete. +// +// The result is verified by parsing it before returning; if it somehow does not +// parse, the caller is told rather than handed a broken snippet. +func buildUsageExample(d WidgetDescription) (example string, omitted []string) { + name := "widget1" + + // Head: the widget's own keyword when the grammar takes it, else the + // explicit-id form. Probed, never assumed. + head := "pluggablewidget '" + d.WidgetID + "' " + name + if kw := strings.ToLower(d.MDLName); kw != "" && widgetKeywordParses(kw) { + head = kw + " " + name + } + + // Scalars: only those whose value can be written as a literal. A datasource, + // attribute, action or expression needs a real name from the project, and + // inventing one would produce an example that parses but cannot run. + var props []string + var needBinding []string + for _, p := range d.Properties { + if p.System { + continue + } + // The two sources spell property types differently — a project .mpk + // gives "datasource", the embedded template "DataSource" — so this + // folds case. Matching only one spelling silently emptied the example + // for every widget described without a project. + // hiddenUnder applies to BOTH branches. It used to gate only the + // binding branch, so the example emitted literals its own configuration + // hides — `heightUnit: 'aspectRatio'` followed by the `height` that + // choice hides — and mxcli's own MDL-WIDGET10 then warned about 32 of + // them. The generator and the checker implement the same editorConfig + // rules; disagreeing is worse than either alone, because the example is + // what a reader copies. + if hiddenUnder(d, p.Key) { + continue + } + switch strings.ToLower(p.Type) { + case "boolean", "integer", "enumeration", "string", "texttemplate": + if p.Required { + props = append(props, " "+p.Key+": "+exampleLiteral(p)) + } + case "attribute", "datasource", "action", "expression", "selection": + if p.Required { + needBinding = append(needBinding, p.Key+" ("+strings.ToLower(p.Type)+")") + } + } + } + + // Names are numbered across the whole body: two widgets sharing a name on + // one page is invalid, and the parser does not catch it — the same defect + // the generated .md had. + var body []string + n := 0 + for _, c := range d.Containers { + if !c.Authorable { + omitted = append(omitted, c.Keyword) + continue + } + n++ + if c.Kind == "child slot" { + body = append(body, fmt.Sprintf(" %s slot%d {\n -- widgets for `%s`\n }", c.Keyword, n, c.PropertyKey)) + continue + } + item := fmt.Sprintf(" %s item%d", c.Keyword, n) + if k, lit := itemExampleLiteral(d, c); k != "" { + item += " (" + k + ": " + lit + ")" + } + body = append(body, item+" -- one entry of `"+c.PropertyKey+"`") + } + + var sb strings.Builder + sb.WriteString(head) + if len(props) > 0 { + sb.WriteString(" (\n" + strings.Join(props, ",\n") + "\n)") + } + if len(body) > 0 { + sb.WriteString(" {\n" + strings.Join(body, "\n") + "\n}") + } + out := sb.String() + + if !pageBodyParses(out) { + return "", append(omitted, "(example could not be generated for this widget)") + } + for _, n := range needBinding { + omitted = append(omitted, n+" — needs a name from your project") + } + return out, omitted +} + +// itemExampleLiteral picks the sub-property to show on an object-list item, and +// a value for it that the validator will accept. +// +// It used to take ItemKeys[0] and write `'…'`. For an ENUMERATION sub-property +// that is simply a wrong value, and mxcli's own MDL-WIDGET08 said so — "property +// `dataSet` has invalid value `…` — valid values are static, dynamic" — on 11 of +// the fixture's examples. The block claims to parse as written, and it did; it +// just did not CHECK as written, which is the more useful promise. +// +// The values were already in hand: propsFromMPK carries an object-list +// property's sub-properties as Children, with their enums and defaults, so the +// same exampleLiteral used for the widget's own scalars applies here. +// +// Preference order is deliberate: a sub-property with a derivable literal (a +// default, or an enumeration's first member) beats one without, because for a +// free-text sub-property there is no correct value to invent and a placeholder +// is the honest output — and the validator accepts any string, so it costs +// nothing. +func itemExampleLiteral(d WidgetDescription, c DescribedContainer) (key, literal string) { + if len(c.ItemKeys) == 0 { + return "", "" + } + children := map[string]DescribedProperty{} + for _, p := range d.Properties { + if !strings.EqualFold(p.Key, c.PropertyKey) { + continue + } + for _, ch := range p.Children { + children[strings.ToLower(ch.Key)] = ch + } + } + // The definition WINS, for the same reason it does in exampleValues: it is + // what MDL-WIDGET08 checks the value against. + for _, it := range c.items { + if it.Default == "" && len(it.Enum) == 0 { + continue + } + children[strings.ToLower(it.Key)] = it + } + for _, k := range c.ItemKeys { + ch, ok := children[strings.ToLower(k)] + if !ok { + continue + } + if ch.Default == "" && len(ch.Enum) == 0 { + continue // nothing to derive; keep looking for one that has something + } + return k, exampleLiteral(ch) + } + // No sub-property offers a value. Show the first key with a placeholder — + // it is a free-text slot, which the validator accepts. + return c.ItemKeys[0], "'…'" +} + +// exampleLiteral picks a writable value for a scalar property: its default when +// it has one, else the first enumeration value, else a placeholder. +func exampleLiteral(p DescribedProperty) string { + switch strings.ToLower(p.Type) { + case "boolean": + if p.Default != "" { + return p.Default + } + return "false" + case "integer": + if p.Default != "" { + return p.Default + } + return "0" + } + if p.Default != "" { + return "'" + p.Default + "'" + } + if len(p.Enum) > 0 { + return "'" + p.Enum[0] + "'" + } + return "'…'" +} + +// widgetKeywordParses reports whether ` name (…)` is accepted as a +// widget in a page body — the head-form half of the same probe the containers use. +func widgetKeywordParses(keyword string) bool { + return pageBodyParses(keyword + " probe1 (someProp: 'x')") +} + +// pageBodyParses puts a fragment in a minimal page and reports whether it parses. +func pageBodyParses(body string) bool { + src := "create page Probe.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + body + "\n}\n" + _, errs := visitor.Build(src) + return len(errs) == 0 +} + +// exampleValues is the configuration the example describes: each scalar +// property's default, which is also what the example writes for the required +// ones. Visibility rules are evaluated against this. +func exampleValues(d WidgetDescription) map[string]string { + values := map[string]string{} + var walk func(props []DescribedProperty) + walk = func(props []DescribedProperty) { + for _, p := range props { + if p.Default != "" { + values[p.Key] = p.Default + } else if len(p.Enum) > 0 { + // An enumeration with no declared default takes its first value, + // which is what Mendix shows in the editor. + values[p.Key] = p.Enum[0] + } + walk(p.Children) + } + } + walk(d.Properties) + // The definition's default WINS over the package's. It is what mxcli writes + // when a script is silent, and therefore what the validator concludes the + // property holds — which is the whole point of resolving it here. + for k, v := range d.defDefaults { + if v != "" { + values[k] = v + } + } + return values +} + +// definitionDefaults collects the value each mapping falls back to, mirroring +// widgetValueMap's own order: an explicit `default`, else a primitive mapping's +// `value` (the widget XML's defaultValue, which is where the generator and the +// checker have to agree). +func definitionDefaults(def *WidgetDefinition) map[string]string { + if def == nil { + return nil + } + out := map[string]string{} + collect := func(mappings []PropertyMapping) { + for _, m := range mappings { + if m.PropertyKey == "" { + continue + } + switch { + case m.Default != "": + out[m.PropertyKey] = m.Default + case m.Operation == "primitive" && m.Value != "": + out[m.PropertyKey] = m.Value + case m.Operation == "selection": + // An omitted `Selection:` is WRITTEN as None — the builder's own + // behaviour, not a guess — and a selection property declares no + // defaultValue in the .mpk, which is why the generator saw + // DataGrid2's `itemSelection` as indeterminable and emitted the + // `itemSelectionMethod` that "None" hides. Same reasoning, and + // same three branches, as widgetValueMap. + out[m.PropertyKey] = "None" + } + if m.Operation == "selection" { + out[m.PropertyKey] = canonicalSelection(out[m.PropertyKey]) + } + } + } + collect(def.PropertyMappings) + for _, mode := range def.Modes { + collect(mode.PropertyMappings) + } + return out +} + +// hiddenUnder reports whether a property is hidden in the configuration the +// example describes, so its binding need not be asked for. +// +// Conservative in the direction of asking too much rather than too little: a +// rule whose condition property has no determinable value does NOT prune, and +// nested rules (about an object-list item) never apply to the widget itself. +// Over-listing a binding costs the reader a moment; hiding one they actually +// need would send them to a build error, which is the failure this whole area +// keeps producing. +func hiddenUnder(d WidgetDescription, propertyKey string) bool { + values := exampleValues(d) + for _, r := range d.Rules { + if r.Nested || r.Cond == nil || !strings.EqualFold(r.Property, propertyKey) { + continue + } + if _, known := values[r.Cond.PropertyKey]; !known { + continue // indeterminable — do not guess, keep asking for it + } + if r.Cond.Hidden(values) { + return true + } + } + return false +} diff --git a/mdl/executor/widget_describe_moved_test.go b/mdl/executor/widget_describe_moved_test.go new file mode 100644 index 0000000000..43691354e3 --- /dev/null +++ b/mdl/executor/widget_describe_moved_test.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +func TestWidgetDescribe_UnknownWidget(t *testing.T) { + reg, err := NewWidgetRegistry() + if err != nil { + t.Fatalf("registry: %v", err) + } + id, _ := resolveWidgetTarget(reg, "NOPE") + if id != "" { + t.Errorf("resolveWidgetTarget(NOPE) = %q, want empty", id) + } + // DATAGRID2 resolves via the builtin alias even without a .def.json entry. + if id, _ := resolveWidgetTarget(reg, "datagrid2"); id != "com.mendix.widget.web.datagrid.Datagrid" { + t.Errorf("resolveWidgetTarget(datagrid2) = %q", id) + } +} + +// TestConditionText renders the four operators as readable English. +func TestConditionText(t *testing.T) { + cases := []struct { + op, val, want string + }{ + {"eq", "None", `itemSelection = "None"`}, + {"ne", "Multi", `itemSelection ≠ "Multi"`}, + {"truthy", "", "itemSelection is set"}, + {"falsy", "", "itemSelection is not set"}, + } + for _, c := range cases { + got := conditionText(&types.WidgetVisibilityCondition{PropertyKey: "itemSelection", Operator: c.op, Value: c.val}) + if got != c.want { + t.Errorf("op %s: got %q, want %q", c.op, got, c.want) + } + } +} diff --git a/mdl/executor/widget_describe_test.go b/mdl/executor/widget_describe_test.go new file mode 100644 index 0000000000..6395ba9a94 --- /dev/null +++ b/mdl/executor/widget_describe_test.go @@ -0,0 +1,372 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// A widget was the only MDL extension point with no in-language DESCRIBE, which +// is why `widget init` had to generate documentation — and why that +// documentation could drift from what the parser accepts. The statement is only +// worth having if it answers without a project, which is the state an agent is +// in when it asks "what can I write here?". +func TestDescribeWidget_AnswersFromEmbeddedKnowledgeWithNoProject(t *testing.T) { + desc, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatalf("DescribeWidget with no project: %v", err) + } + if desc.WidgetID != "com.mendix.widget.web.combobox.Combobox" { + t.Errorf("WidgetID = %q", desc.WidgetID) + } + if desc.Source != "embedded template" { + t.Errorf("Source = %q, want the embedded fallback", desc.Source) + } + if len(desc.Properties) == 0 { + t.Error("no properties — an empty description answers nothing") + } +} + +// The full widget id must work as well as the MDL keyword: it is what a widget +// package, a page's BSON and the generated docs all carry, and for a widget with +// no keyword it is the only name there is. +func TestDescribeWidget_AcceptsTheWidgetIdAsWellAsTheKeyword(t *testing.T) { + byKeyword, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatal(err) + } + byID, err := DescribeWidget("com.mendix.widget.web.combobox.Combobox", "") + if err != nil { + t.Fatal(err) + } + if byKeyword.WidgetID != byID.WidgetID { + t.Errorf("keyword gave %q, id gave %q", byKeyword.WidgetID, byID.WidgetID) + } + if len(byKeyword.Properties) != len(byID.Properties) { + t.Errorf("property counts differ: %d vs %d", len(byKeyword.Properties), len(byID.Properties)) + } +} + +// An unknown widget must say so rather than returning an empty description that +// reads as "this widget has no properties". +func TestDescribeWidget_UnknownWidgetIsAnError(t *testing.T) { + if _, err := DescribeWidget("notawidget", ""); err == nil { + t.Fatal("want an error for an unknown widget, got none") + } +} + +// The project's installed .mpk is preferred over the embedded template, because +// it is version-accurate and is the only place a Marketplace widget appears. +// This is the control for the no-project test above: without it, "embedded +// template" there is equally consistent with the .mpk path never running. +func TestDescribeWidget_PrefersTheProjectPackageOverTheEmbeddedTemplate(t *testing.T) { + desc, err := DescribeWidget("combobox", "../../testdata/expr-checker/minimal.mpr") + if err != nil { + t.Skipf("fixture unavailable: %v", err) + } + if desc.Source != "project .mpk" { + t.Errorf("Source = %q, want the project package to win", desc.Source) + } +} + +// The rendered form is what a reader actually sees, and it is shared with +// `mxcli widget describe` — so a change that broke it would break both. +func TestPrintWidgetDescription_RendersTheHeaderAndProperties(t *testing.T) { + desc, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatal(err) + } + var sb strings.Builder + PrintWidgetDescription(&sb, *desc) + out := sb.String() + for _, want := range []string{"Widget:", "ID:", "Kind:", "Properties ("} { + if !strings.Contains(out, want) { + t.Errorf("rendered output missing %q:\n%s", want, out) + } + } +} + +// The containers are the half of a widget's shape that DESCRIBE WIDGET was +// missing relative to the generated .md — and the half whose MDL syntax is +// currently wrong for most widgets, so reporting them without saying which are +// reachable would repeat the .md's mistake. +// +// Gallery is used because it is an EMBEDDED definition carrying containers on +// both sides of the answer, so this needs no project and cannot skip. An +// earlier version pointed at the fixture project and skipped every run, since +// the fixture has no extracted defs — a test that only ever skips proves +// nothing (see #808 in fix-issue.md). +func TestDescribeWidget_ReportsContainersAndWhetherTheyAreAuthorable(t *testing.T) { + desc, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatalf("DescribeWidget(gallery): %v", err) + } + byKeyword := map[string]DescribedContainer{} + for _, c := range desc.Containers { + byKeyword[c.Keyword] = c + } + if len(byKeyword) == 0 { + t.Fatal("no containers reported for a widget that has three") + } + + // Both of these used to sit on opposite sides of the answer: `template` was + // in the grammar's container vocabulary and `emptyplaceholder` was not. + // Slices 2-3 removed that boundary, so both are authorable now — measured + // across the fixture's definitions, 50 of 50 containers are. + // + // The assertion is kept pointing at the SAME two containers deliberately. + // It is the regression test for the capability: if `emptyplaceholder` ever + // reports unauthorable again, the def-driven body has been lost. + for _, kw := range []string{"emptyplaceholder", "template"} { + c, ok := byKeyword[kw] + if !ok { + t.Errorf("%s missing; got %v", kw, desc.Containers) + continue + } + if !c.Authorable { + t.Errorf("%s reported unauthorable — since slices 2-3 every container a definition "+ + "declares can be written (mendixlabs/mxcli#1036)", kw) + } + } +} + +// Authorability is derived by parsing, never from a list. A list here would be +// the same defect the whole proposal is about, one layer up — so an invented +// keyword must come back false through the same path a real one comes back true. +func TestContainerKeywordParses_DerivesTheAnswerRatherThanListingIt(t *testing.T) { + if !containerKeywordParses("group", false) { + t.Error("group should parse as an object-list container") + } + // An invented keyword now PARSES — that is what slices 2-3 did, and it is + // why the wrong-name check moved to the validator (MDL-WIDGET25/26), which + // can consult the parent's definition where the parser cannot. + if !containerKeywordParses("definitelynotakeyword", false) { + t.Error("since slice 3 any name parses in a container position; the check that it is a " + + "REAL container belongs to MDL-WIDGET26, not to the parser") + } + // The probe must still be a real probe. A body that is malformed for a + // reason unrelated to the keyword has to come back false, or "authorable" + // would be a constant dressed up as a derivation — the exact defect this + // test exists to prevent, one layer up. + if containerKeywordParses("group (", false) { + t.Error("a malformed probe reported authorable — the parse probe is not actually running") + } + if containerKeywordParses("", false) { + t.Error("an empty keyword must not report as authorable") + } +} + +// The example is the half of the generated .md that was WRONG — its version +// promised syntax that failed on its own first line. This one is built from +// probes, so the guarantee is testable: feed it back and it parses. +func TestDescribeWidget_ExampleParsesAsWritten(t *testing.T) { + for _, w := range []string{"gallery", "combobox", "image"} { + desc, err := DescribeWidget(w, "") + if err != nil { + t.Fatalf("DescribeWidget(%s): %v", w, err) + } + if desc.Example == "" { + t.Errorf("%s: no example emitted", w) + continue + } + if !pageBodyParses(desc.Example) { + t.Errorf("%s: emitted example does not parse:\n%s", w, desc.Example) + } + } +} + +// A container MDL cannot express must be left out AND named. Silently including +// it is what made the .md misleading rather than merely incomplete; silently +// dropping it would be almost as bad, since the reader would never learn the +// widget has it. +func TestDescribeWidget_ExampleOmitsWhatItCannotFillAndSaysSo(t *testing.T) { + desc, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatal(err) + } + + // Since slices 2-3 no container is omitted for being unwritable — all three + // of Gallery's appear. This half is the capability regression test: it fails + // if the def-driven body is lost. + for _, kw := range []string{"emptyplaceholder", "filter", "template"} { + if !strings.Contains(desc.Example, kw) { + t.Errorf("container %q missing from the example — every container a definition declares "+ + "should now be writable:\n%s", kw, desc.Example) + } + } + + // The omission machinery still has a job: a BINDING cannot be invented, + // because it needs a name from the reader's own project. Those must be left + // out and named, which is what stopped the generated .md from promising + // syntax that failed. + if len(desc.OmittedFromExample) == 0 { + t.Fatal("nothing reported as omitted — Gallery's datasource cannot be filled in, " + + "so the example must say so rather than invent one") + } + var named bool + for _, o := range desc.OmittedFromExample { + if strings.Contains(strings.ToLower(o), "datasource") { + named = true + } + } + if !named { + t.Errorf("the unfillable datasource is not named among the omissions; got %v", desc.OmittedFromExample) + } + if strings.Contains(desc.Example, "DataSource:") { + t.Errorf("the example invented a datasource instead of omitting it:\n%s", desc.Example) + } +} + +// Two widgets sharing a name on one page is invalid, and the parser does not +// catch it — so the example has to number them itself. The .md generator had +// this same defect. +func TestDescribeWidget_ExampleNamesAreUnique(t *testing.T) { + desc, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatal(err) + } + seen := map[string]bool{} + for _, line := range strings.Split(desc.Example, "\n") { + fields := strings.Fields(strings.TrimSpace(line)) + if len(fields) < 2 { + continue + } + name := fields[1] + if !strings.HasPrefix(name, "slot") && !strings.HasPrefix(name, "item") { + continue + } + if seen[name] { + t.Errorf("duplicate widget name %q in example:\n%s", name, desc.Example) + } + seen[name] = true + } + if len(seen) < 2 { + t.Skip("widget has fewer than two named containers; nothing to collide") + } +} + +// The head form is probed too: a widget whose keyword the grammar accepts uses +// it, and one whose keyword it does not falls back to the explicit-id form. +// Both halves in one test, so neither can pass vacuously. +func TestDescribeWidget_ExampleHeadFormFollowsTheGrammar(t *testing.T) { + authorable, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(authorable.Example, "gallery widget1") { + t.Errorf("gallery's keyword parses, so the example should use it:\n%s", authorable.Example) + } + + notYet, err := DescribeWidget("image", "") + if err != nil { + t.Fatal(err) + } + if strings.HasPrefix(notYet.Example, "pluggablewidget") == strings.HasPrefix(notYet.Example, "image widget1") { + t.Fatalf("indeterminate head form:\n%s", notYet.Example) + } +} + +// A widget's properties are "required" only where the editor shows them. Combo +// box declares eleven bindings across mutually exclusive options-source modes, +// so listing them all overstates what a reader must supply — the same class of +// misinformation as the generated .md, quieter: not syntax that fails, but work +// that is not needed. +// +// Needs a project: the rules come from the .mpk's editorConfig, so the embedded +// template carries none. Asserted below rather than left to be discovered. +func TestDescribeWidget_BindingsHiddenUnderTheExampleAreNotAskedFor(t *testing.T) { + const fixture = "../../testdata/expr-checker/minimal.mpr" + desc, err := DescribeWidget("combobox", fixture) + if err != nil { + t.Fatalf("DescribeWidget(combobox, fixture): %v", err) + } + if len(desc.Rules) == 0 { + t.Fatal("no visibility rules from the project .mpk — nothing could be pruned, so this would pass vacuously") + } + omitted := strings.Join(desc.OmittedFromExample, "; ") + + // attributeBoolean is hidden when optionsSourceType ≠ "boolean", and the + // default is "association" — so the rule fires and it must not be asked for. + if strings.Contains(omitted, "attributeBoolean") { + t.Errorf("attributeBoolean is hidden under the example's configuration but was still asked for:\n%s", omitted) + } +} + +// The control. Pruning must not simply drop every binding: a widget whose +// datasource nothing hides still has to ask for it, or the example would look +// complete while being unusable. +func TestDescribeWidget_VisibleBindingsAreStillAskedFor(t *testing.T) { + for _, w := range []string{"gallery", "datagrid"} { + desc, err := DescribeWidget(w, "") + if err != nil { + t.Fatalf("%s: %v", w, err) + } + if !strings.Contains(strings.Join(desc.OmittedFromExample, "; "), "datasource") { + t.Errorf("%s: its datasource is not hidden by any rule, so it must still be asked for; got %v", + w, desc.OmittedFromExample) + } + } +} + +// Conservatism, stated as a test: a rule whose condition property has no +// determinable value must NOT prune. Over-listing costs the reader a moment; +// hiding a binding they actually need sends them to a build error, which is the +// failure this area keeps producing. +func TestHiddenUnder_DoesNotPruneOnAnIndeterminableCondition(t *testing.T) { + d := WidgetDescription{ + Properties: []DescribedProperty{ + {Key: "someBinding", Type: "datasource", Required: true}, + // `mode` has no default and no enum, so its value is unknowable. + {Key: "mode", Type: "enumeration"}, + }, + Rules: []DescribedRule{{ + Property: "someBinding", + Cond: &types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "ne", Value: "x"}, + }}, + } + if hiddenUnder(d, "someBinding") { + t.Error("pruned on a condition whose value cannot be determined") + } + + // The control: give `mode` a default the condition matches, and it prunes. + d.Properties[1].Default = "y" // "y" != "x", so `ne` fires + if !hiddenUnder(d, "someBinding") { + t.Error("did not prune when the condition is determinable and fires") + } +} + +// A rule about an object-list ITEM's property must never prune the WIDGET's +// binding of the same name — they are different properties on different objects. +func TestHiddenUnder_IgnoresNestedItemRules(t *testing.T) { + d := WidgetDescription{ + Properties: []DescribedProperty{ + {Key: "caption", Type: "attribute", Required: true}, + {Key: "mode", Type: "enumeration", Default: "y"}, + }, + Rules: []DescribedRule{{ + Property: "caption", + Nested: true, + Cond: &types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "ne", Value: "x"}, + }}, + } + if hiddenUnder(d, "caption") { + t.Error("a nested item rule pruned the widget's own binding") + } +} + +// The limitation behind the test above, stated so it is not rediscovered: with +// no project there is no .mpk, so no editorConfig, so no rules — and nothing to +// prune with. The description is still useful, it just cannot narrow the +// bindings. +func TestDescribeWidget_NoProjectMeansNoVisibilityRules(t *testing.T) { + desc, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatal(err) + } + if len(desc.Rules) != 0 { + t.Errorf("embedded combobox unexpectedly carries %d rules; the pruning test's project requirement may be stale", len(desc.Rules)) + } +} diff --git a/mdl/executor/widget_describe_validator_agreement_test.go b/mdl/executor/widget_describe_validator_agreement_test.go new file mode 100644 index 0000000000..e06de80582 --- /dev/null +++ b/mdl/executor/widget_describe_validator_agreement_test.go @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +// fixtureProject is a real project with widgets/*.mpk committed. The .def.json +// CACHE under .mxcli/ is gitignored, so a test must not depend on it — which is +// the point here: the nine hand-crafted widgets never get a .def.json anyway, +// and they are exactly the ones this test is about. +func fixtureProject(t *testing.T) string { + t.Helper() + p, err := filepath.Abs(filepath.Join("..", "..", "testdata", "expr-checker", "minimal.mpr")) + if err != nil { + t.Skipf("cannot resolve fixture: %v", err) + } + if _, err := os.Stat(p); err != nil { + t.Skipf("fixture project not present: %v", err) + } + return p +} + +// DESCRIBE WIDGET and the property validator must agree about which properties a +// widget has. They read different sources — DESCRIBE reads the project's +// installed .mpk, the validator reads the WidgetDefinition — and they disagreed. +// +// # The defect +// +// Nine widgets (combobox, gallery, image, barcodescanner, the four data-grid +// filters, dropdownsort) have hand-crafted definitions in sdk/widgets/definitions/ +// and are deliberately never extracted per-project, so no .def.json is ever +// generated for them. Those hand-written definitions cover a fraction of the +// widget: +// +// combobox 73 properties in the .mpk, 7 mapped + 4 known +// gallery 44 12 +// image 37 14 +// barcodescanner 12 1 +// +// So `DESCRIBE WIDGET` emitted an example it labels "parses as written" — and it +// does parse — naming 33 properties that mxcli's OWN validator then rejected with +// MDL-WIDGET01. Since exec refuses before writing, barcodescanner could not be +// placed from MDL at all: include the five properties and exec refuses, omit them +// and mxbuild reports CE0463 "the definition of this widget has changed". +// Combobox was the sharpest case — its describe output marks `source` REQUIRED +// and its validator said the widget has no such property. +// +// Reported by an external test project against 41c55d09 + this PR, retested at +// bca5466e, and reproduced here at 64055caa: combobox 17, gallery 7, +// barcodescanner 5, image 4. +// +// # Why this is the right assertion +// +// Not "the example parses" — it already did. The generator and the validator are +// two readers of one widget, so the invariant is that a property one of them +// emits is not one the other calls nonexistent. +func TestDescribeWidgetPropertiesAreAcceptedByTheValidator(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + // The nine hand-crafted widgets are the population this is about; assert the + // fixture actually has some of them installed, or the test proves nothing. + targets := []string{"combobox", "gallery", "image", "barcodescanner"} + var checked int + var bad []string + + for _, name := range targets { + desc, err := DescribeWidget(name, project) + if err != nil { + continue + } + if desc.Source != "project .mpk" { + // Falling back to the embedded template means the .mpk is absent, and + // then both sides read the same thin data and cannot disagree. + continue + } + def, ok := registry.Get(name) + if !ok || def == nil { + continue + } + checked++ + + allowed, _ := allowedWidgetProperties(def) + known := knownUnmappedProperties(def, allowed) + for _, p := range desc.Properties { + if p.Key == "" || isSystemPropKey(p.Key) { + continue + } + k := strings.ToLower(p.Key) + if allowed[k] || known[k] { + continue + } + bad = append(bad, name+"."+p.Key) + } + } + + if checked == 0 { + t.Skip("none of the hand-crafted widgets are installed in the fixture with a .mpk") + } + if len(bad) > 0 { + sort.Strings(bad) + t.Errorf("%d properties DESCRIBE WIDGET emits are rejected by the validator "+ + "(MDL-WIDGET01 \"has no property\"); the two read different sources and must not "+ + "disagree:\n %s", len(bad), strings.Join(bad, "\n ")) + } +} + +// The control: enrichment must not make the validator accept ANYTHING. A property +// no widget declares is still an error, or MDL-WIDGET01 stops detecting typos — +// which is the rule's whole job. +func TestValidatorStillRejectsAPropertyTheMPKDoesNotDeclare(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + def, ok := registry.Get("combobox") + if !ok || def == nil { + t.Skip("combobox not in registry") + } + allowed, _ := allowedWidgetProperties(def) + known := knownUnmappedProperties(def, allowed) + + for _, bogus := range []string{"notarealproperty", "sourceX", "optionsSourceTypo"} { + k := strings.ToLower(bogus) + if allowed[k] || known[k] { + t.Errorf("%q was accepted; enrichment must add the widget's REAL properties, "+ + "not open the gate", bogus) + } + } +} diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index c42695c59b..f0b4a1e54f 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -329,8 +329,14 @@ func (e *PluggableWidgetEngine) Build(def *WidgetDefinition, w *ast.WidgetV3) (* // `associatedFiles` and `associatedImages` from `DataSource:`, gated on // `uploadMode`). Naming a hidden property outright is still an error, raised // by MDL-WIDGET10 at check time rather than silently dropped here (#956). + // + // builder.PrimitiveValues() is read HERE, before any mapping is applied, so + // it is the template's captured configuration — the values an unmapped + // property will actually be stored with. A rule keyed on one of those has to + // be evaluated against them; see hiddenUnnamedProperties. hiddenSkip := e.hiddenUnnamedProperties(def, w, - widgetPropertyDefaults(e.pageBuilder.getProjectPath(), def.WidgetID)) + widgetPropertyDefaults(e.pageBuilder.getProjectPath(), def.WidgetID), + builder.PrimitiveValues()) for _, mapping := range mappings { if reset, hidden := hiddenSkip[strings.ToLower(mapping.PropertyKey)]; hidden { if reset != "" && mapping.Operation == "primitive" { @@ -640,9 +646,13 @@ func (e *PluggableWidgetEngine) isPrimaryAttributeMapping(mapping PropertyMappin // source MDL-WIDGET10 reads, so the writer and the checker cannot disagree about // what a default is, which is how this shipped. // +// stored is the TEMPLATE's captured configuration, read before any mapping was +// applied. It is what answers a rule keyed on a property MDL cannot name; see +// the fallback chain below. +// // Rules come from the .def.json, falling back to a live lift from the installed // .mpk (the same two sources the visibility application uses). -func (e *PluggableWidgetEngine) hiddenUnnamedProperties(def *WidgetDefinition, w *ast.WidgetV3, defaults map[string]string) map[string]string { +func (e *PluggableWidgetEngine) hiddenUnnamedProperties(def *WidgetDefinition, w *ast.WidgetV3, defaults, stored map[string]string) map[string]string { rules := e.visibilityRules(def) if len(rules) == 0 { return nil @@ -659,7 +669,7 @@ func (e *PluggableWidgetEngine) hiddenUnnamedProperties(def *WidgetDefinition, w } condKey := strings.ToLower(rule.HiddenWhen.PropertyKey) condVal, known := values[condKey] - if !known { + if !known || condVal == "" { // widgetValueMap only knows properties the definition MAPS, because a // mapping is what gives a property an MDL keyword. A rule whose // condition is an UNMAPPED property was therefore always @@ -668,12 +678,21 @@ func (e *PluggableWidgetEngine) hiddenUnnamedProperties(def *WidgetDefinition, w // produced the one differing field in 1480 on a copied Atlas layout // (mxcli-ledger §142). // - // The declared default is the right fallback for exactly the reason - // the rest of this function exists: MDL cannot name the property, so - // nothing has moved it off its default. It is a fallback and not a - // preference — a value the script set, or a mapping's own default, - // still wins above. - condVal, known = defaults[defaultsKey("", rule.HiddenWhen.PropertyKey)] + // The TEMPLATE's captured value is the answer, and the declared + // default is NOT: the question is whether the property will be hidden + // in the document about to be WRITTEN, and an unmapped property is + // written with whatever the template holds. The two differ exactly + // where it matters — Image declares `maxHeightUnit` "pixels" and the + // template captured "none", so reading the default concluded + // "visible" and left `maxHeight` at the template's 0 against a + // declared 250, which is the whole of §142's CE0463. Both are + // fallbacks and not preferences: a value the script set, or a + // mapping's own default, still wins above. + if v, ok := stored[rule.HiddenWhen.PropertyKey]; ok && v != "" { + condVal, known = v, true + } else { + condVal, known = defaults[defaultsKey("", rule.HiddenWhen.PropertyKey)] + } if !known || condVal == "" { continue // still indeterminable — never guess } diff --git a/mdl/executor/widget_example_hidden_props_test.go b/mdl/executor/widget_example_hidden_props_test.go new file mode 100644 index 0000000000..2504c2ef3e --- /dev/null +++ b/mdl/executor/widget_example_hidden_props_test.go @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "regexp" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +var examplePropLine = regexp.MustCompile(`(?m)^\s{2}([A-Za-z][A-Za-z0-9_]*)\s*:`) + +// exampleScalarKeys returns the property keys the generated example writes in the +// widget's own head — the ` key: value,` lines, not container item properties. +func exampleScalarKeys(example string) []string { + head := example + if i := strings.Index(head, ") {"); i >= 0 { + head = head[:i] + } + var out []string + for _, m := range examplePropLine.FindAllStringSubmatch(head, -1) { + out = append(out, m[1]) + } + return out +} + +// The generator narrows its example by the widget's editorConfig hide-rules, and +// the validator implements the same rules as MDL-WIDGET10. They disagreed: 32 +// warnings over the fixture's widgets, every one of them naming a property the +// example ITSELF had just emitted. +// +// videoplayer example: heightUnit: 'aspectRatio', … height: 500 +// validator: property `height` is hidden when `heightUnit` is +// "aspectRatio" — the value will be ignored +// +// The generator chose `heightUnit: 'aspectRatio'` and then wrote the `height` +// its own rule hides. Cause: hiddenUnder was consulted only on the branch that +// asks for a BINDING (attribute/datasource/action/expression/selection) and not +// on the scalar branch that emits literals — so the narrowing existed and half +// the properties skipped it. +// +// Reported by an external test project (14 of 14 warnings were self-inflicted +// there); reproduced here at 668ad9ae over all 42 definitions. +func TestUsageExampleDoesNotEmitItsOwnHiddenProperties(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var offenders []string + var checked int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + checked++ + for _, key := range exampleScalarKeys(desc.Example) { + if hiddenUnder(*desc, key) { + offenders = append(offenders, def.MDLName+"."+key) + } + } + } + if checked == 0 { + t.Skip("no widgets described") + } + if len(offenders) > 0 { + t.Errorf("%d properties are emitted by the example and hidden by that same "+ + "example's configuration — the generator and MDL-WIDGET10 must not disagree:\n %s", + len(offenders), strings.Join(offenders, "\n ")) + } +} + +// The control. "Emit nothing hidden" is trivially satisfied by emitting nothing, +// and it is also satisfied if hiddenUnder always returns false — in which case +// the test above proves nothing at all. Assert that pruning REALLY fires: some +// widget must have a scalar property that hiddenUnder reports hidden under the +// example's own configuration, i.e. something was actually removed. +func TestUsageExamplePruningActuallyFires(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var pruned int + var nonEmpty int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + if len(exampleScalarKeys(desc.Example)) > 0 { + nonEmpty++ + } + for _, p := range desc.Properties { + if p.System || !p.Required { + continue + } + switch strings.ToLower(p.Type) { + case "boolean", "integer", "enumeration", "string", "texttemplate": + if hiddenUnder(*desc, p.Key) { + pruned++ + } + } + } + } + if nonEmpty == 0 { + t.Fatal("no example emits any property — the first test would pass vacuously") + } + if pruned == 0 { + t.Fatal("hiddenUnder never fires on a required scalar, so the assertion in " + + "TestUsageExampleDoesNotEmitItsOwnHiddenProperties is vacuous") + } + t.Logf("%d required scalars pruned across %d widgets with a non-empty example", pruned, nonEmpty) +} + +// The end-to-end form of the same invariant, and the one that matches how the +// disagreement was reported: build each widget's example, parse it into a real +// page statement, and run the property validator over it. Zero MDL-WIDGET10. +// +// This is stronger than the structural test above, which can only see what the +// generator itself considers hidden. The residue it catches is the OTHER +// direction of the same split: the validator resolves a property's value from +// the widget DEFINITION's mapping defaults (gallery's `itemSelection` defaults +// to "Single", so `keepSelection` is hidden), while exampleValues read only the +// .mpk, where a selection property carries no defaultValue — so the generator +// called it indeterminable and emitted the property the validator then warned +// about. Two value sources for one question. +func TestGeneratedExamplesProduceNoHiddenPropertyWarnings(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var offenders []string + var validated int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + src := "create page Probe.P_" + def.MDLName + + " (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + desc.Example + "\n}\n" + prog, errs := visitor.Build(src) + if len(errs) > 0 || prog == nil { + continue // the example-parses guarantee is a different test's business + } + for _, stmt := range prog.Statements { + for _, v := range ValidateWidgetPropertiesForStatement(stmt, registry) { + if v.RuleID == "MDL-WIDGET10" { + offenders = append(offenders, def.MDLName+": "+v.Message) + } + } + } + validated++ + } + + if validated == 0 { + t.Skip("no example validated") + } + if len(offenders) > 0 { + t.Errorf("%d hidden-property warnings on mxcli's OWN generated examples "+ + "(%d widgets validated) — the generator and MDL-WIDGET10 read the same "+ + "editorConfig rules and must reach the same answer:\n %s", + len(offenders), validated, strings.Join(offenders, "\n ")) + } +} diff --git a/mdl/executor/widget_example_item_literal_test.go b/mdl/executor/widget_example_item_literal_test.go new file mode 100644 index 0000000000..ab8ee89728 --- /dev/null +++ b/mdl/executor/widget_example_item_literal_test.go @@ -0,0 +1,189 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// The example writes one sub-property on each object-list item to show the +// shape. It wrote `'…'` — a literal ellipsis — and mxcli's own MDL-WIDGET08 +// then rejected it: +// +// widget `item2` (series) property `dataSet` has invalid value `…` +// — valid values are static, dynamic +// +// 11 across the fixture's 42 widgets. `'…'` reads as a placeholder to a person +// and is simply a wrong value to the checker, so the example was not runnable +// as printed even though it parsed — and "parses as written" is exactly what +// the block claims. +// +// The values are already in hand: propsFromMPK carries an object-list +// property's item properties as Children, with their enums and defaults, so +// exampleLiteral can pick a real one from the same data. +// +// The assertion is "no MDL-WIDGET08", NOT "no ellipsis anywhere". A free-text +// sub-property has no correct value to invent, and the validator accepts any +// string, so a placeholder there is the honest output — narrowing the rule to +// what the checker actually rejects keeps the test about the defect rather than +// about a character. +func TestGeneratedExamplesUseRealItemValues(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var offenders []string + var validated, withItems int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + src := "create page Probe.P_" + def.MDLName + + " (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + desc.Example + "\n}\n" + prog, errs := visitor.Build(src) + if len(errs) > 0 || prog == nil { + continue + } + for _, stmt := range prog.Statements { + for _, v := range ValidateWidgetPropertiesForStatement(stmt, registry) { + if v.RuleID == "MDL-WIDGET08" { + offenders = append(offenders, def.MDLName+": "+v.Message) + } + } + } + validated++ + for _, c := range desc.Containers { + if c.Kind == "object list" && c.Authorable && len(c.ItemKeys) > 0 { + withItems++ + } + } + } + + // Without this the assertion is satisfied by a build where no example has an + // object list at all — so when that is the case, this test proves nothing and + // says so rather than passing quietly. + // + // It SKIPS rather than fails, because the empty case is legitimate and is + // exactly what CI sees: .mxcli/widgets/*.def.json is derived and gitignored, + // so a fresh checkout has only the hand-crafted definitions in + // sdk/widgets/definitions/, none of which declares an authorable object list + // with item properties. The guarantee comes from + // TestItemExampleLiteral_* below, which is hermetic and runs everywhere; + // this test is the end-to-end confirmation where the environment can give it. + if withItems == 0 { + t.Skip("no authorable object list with item properties in this environment " + + "(the .def.json cache is gitignored, so CI has only the hand-crafted " + + "definitions) — see TestItemExampleLiteral_* for the hermetic assertion") + } + if len(offenders) > 0 { + t.Errorf("%d generated examples carry a placeholder value the validator rejects "+ + "(%d validated, %d authorable object lists):\n %s", + len(offenders), validated, withItems, strings.Join(offenders, "\n ")) + } +} + +func firstLineContaining(s, needle string) string { + for _, line := range strings.Split(s, "\n") { + if strings.Contains(line, needle) { + return strings.TrimSpace(line) + } + } + return "" +} + +// The hermetic half: itemExampleLiteral's contract, asserted on a synthetic +// description so it runs in CI as well as locally. +// +// The end-to-end test above cannot carry this on its own — it needs a widget +// definition with an authorable object list, and CI has none, because the +// .def.json cache those come from is derived and gitignored. A test that only +// runs on a developer's machine is not a guard. +func TestItemExampleLiteral_PrefersASubPropertyWithADerivableValue(t *testing.T) { + desc := WidgetDescription{ + Properties: []DescribedProperty{{ + Key: "series", Type: "object", + Children: []DescribedProperty{ + {Key: "staticName", Type: "texttemplate"}, + {Key: "dataSet", Type: "enumeration", Enum: []string{"static", "dynamic"}}, + }, + }}, + } + c := DescribedContainer{ + Keyword: "series", PropertyKey: "series", Kind: "object list", + ItemKeys: []string{"staticName", "dataSet"}, + } + + key, lit := itemExampleLiteral(desc, c) + if key != "dataSet" { + t.Errorf("key = %q, want dataSet — a sub-property with a derivable value must be "+ + "preferred over a free-text one, or the example shows `'…'` where a real "+ + "member was available", key) + } + if lit != "'static'" { + t.Errorf("literal = %q, want 'static' (the enumeration's first member); `'…'` is "+ + "what MDL-WIDGET08 rejects", lit) + } +} + +// The definition wins over the .mpk, because the definition is what +// MDL-WIDGET08 checks the value against. Measured cause: ParseMPKForWidget +// returns 0 children for a PopupMenu's `basicItems`, while its definition +// carries itemType with enumValues [item, divider]. +func TestItemExampleLiteral_DefinitionBeatsThePackage(t *testing.T) { + desc := WidgetDescription{ + Properties: []DescribedProperty{{ + Key: "basicItems", Type: "object", + // The package knows the key but nothing about its values. + Children: []DescribedProperty{{Key: "itemType", Type: "enumeration"}}, + }}, + } + c := DescribedContainer{ + Keyword: "item", PropertyKey: "basicItems", Kind: "object list", + ItemKeys: []string{"itemType"}, + items: []DescribedProperty{ + {Key: "itemType", Type: "primitive", Default: "item", Enum: []string{"item", "divider"}}, + }, + } + + key, lit := itemExampleLiteral(desc, c) + if key != "itemType" || lit != "'item'" { + t.Errorf("got (%q, %q), want (itemType, 'item') — the definition carries the value "+ + "the package omits, and it is the source the validator reads", key, lit) + } +} + +// A container whose sub-properties are ALL free text keeps the placeholder. +// There is no correct value to invent, the validator accepts any string, and +// inventing one would be worse than admitting the gap. +func TestItemExampleLiteral_FreeTextKeepsThePlaceholder(t *testing.T) { + desc := WidgetDescription{ + Properties: []DescribedProperty{{ + Key: "attributes", Type: "object", + Children: []DescribedProperty{{Key: "attributeName", Type: "string"}}, + }}, + } + c := DescribedContainer{ + Keyword: "attribute", PropertyKey: "attributes", Kind: "object list", + ItemKeys: []string{"attributeName"}, + } + key, lit := itemExampleLiteral(desc, c) + if key != "attributeName" || lit != "'…'" { + t.Errorf("got (%q, %q), want (attributeName, '…')", key, lit) + } +} + +// No sub-properties at all: nothing to write, and no panic on the empty slice. +func TestItemExampleLiteral_NoItemKeys(t *testing.T) { + if key, lit := itemExampleLiteral(WidgetDescription{}, DescribedContainer{}); key != "" || lit != "" { + t.Errorf("got (%q, %q), want empty", key, lit) + } +} diff --git a/mdl/executor/widget_hidden_reset_test.go b/mdl/executor/widget_hidden_reset_test.go index 1ad9d70a01..f05264a0df 100644 --- a/mdl/executor/widget_hidden_reset_test.go +++ b/mdl/executor/widget_hidden_reset_test.go @@ -90,7 +90,7 @@ func imageDefaults() map[string]string { // captured. func TestHiddenProperties_ResetToTheirDeclaredDefault(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} - hidden := e.hiddenUnnamedProperties(imageDef(), &ast.WidgetV3{Name: "img"}, imageDefaults()) + hidden := e.hiddenUnnamedProperties(imageDef(), &ast.WidgetV3{Name: "img"}, imageDefaults(), nil) for _, key := range []string{"width", "height"} { reset, ok := hidden[key] @@ -110,7 +110,7 @@ func TestHiddenProperties_VisiblePropertyIsNotReset(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} w := &ast.WidgetV3{Name: "img", Properties: map[string]any{"WidthUnit": "pixels", "Width": "48"}} - hidden := e.hiddenUnnamedProperties(imageDef(), w, imageDefaults()) + hidden := e.hiddenUnnamedProperties(imageDef(), w, imageDefaults(), nil) if _, ok := hidden["width"]; ok { t.Error("width is visible when widthUnit is pixels — resetting it would discard Width: 48") } @@ -127,7 +127,7 @@ func TestHiddenProperties_ExplicitlyNamedIsLeftToTheChecker(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} w := &ast.WidgetV3{Name: "img", Properties: map[string]any{"Width": "48"}} - if _, ok := e.hiddenUnnamedProperties(imageDef(), w, imageDefaults())["width"]; ok { + if _, ok := e.hiddenUnnamedProperties(imageDef(), w, imageDefaults(), nil)["width"]; ok { t.Error("a hidden property the script named must not be silently reset — MDL-WIDGET10 reports it") } } @@ -137,7 +137,7 @@ func TestHiddenProperties_ExplicitlyNamedIsLeftToTheChecker(t *testing.T) { // old behaviour and remains the fallback. func TestHiddenProperties_NoDeclaredDefaultMeansNoReset(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} - hidden := e.hiddenUnnamedProperties(imageDef(), &ast.WidgetV3{Name: "img"}, nil) + hidden := e.hiddenUnnamedProperties(imageDef(), &ast.WidgetV3{Name: "img"}, nil, nil) reset, ok := hidden["width"] if !ok { @@ -156,7 +156,7 @@ func TestHiddenProperties_DataSourcePruningIsUnchanged(t *testing.T) { def := uploadModeDef() filesMode := e.hiddenUnnamedProperties(def, &ast.WidgetV3{ - Name: "fu", Properties: map[string]any{"DataSource": "assoc"}}, nil) + Name: "fu", Properties: map[string]any{"DataSource": "assoc"}}, nil, nil) if _, ok := filesMode["associatedimages"]; !ok { t.Error("associatedImages was not pruned under the default uploadMode — this is #956's CE0463") } @@ -178,7 +178,7 @@ func TestHiddenProperties_WriterAndCheckerShareTheDefaultsSource(t *testing.T) { } e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} hidden := e.hiddenUnnamedProperties(imageDef(), &ast.WidgetV3{Name: "img"}, - widgetPropertyDefaults("", "com.mendix.widget.web.image.Image")) + widgetPropertyDefaults("", "com.mendix.widget.web.image.Image"), nil) if reset := hidden["width"]; reset != "" { t.Errorf("reset = %q, want empty when the defaults source has nothing", reset) } diff --git a/mdl/executor/widget_hidden_reset_unmapped_test.go b/mdl/executor/widget_hidden_reset_unmapped_test.go index ac5f451c6f..1b441c0327 100644 --- a/mdl/executor/widget_hidden_reset_unmapped_test.go +++ b/mdl/executor/widget_hidden_reset_unmapped_test.go @@ -35,6 +35,32 @@ import ( // editorConfig, and the set MDL can name is decided by mxcli. Making one a subset // of the other was the mistake: every hidden property with a declared default is // written, whether or not MDL has a word for it. +// +// --------------------------------------------------------------------------- +// +// THAT FIX SHIPPED AND CHANGED NOTHING, for a reason worth stating plainly: it +// was verified against an input that does not exist. The helper below used to +// declare `maxHeightUnit`'s default as "none"; Image 1.6.0 declares "pixels" — +// so the test asked the rule a question the real package never asks. +// Re-measured against the .mpk in a live 11.14.0 project: +// +// maxheight = "250" maxheightunit = "pixels" +// +// With "pixels" the condition `maxHeightUnit = none` is FALSE, maxHeight reads +// as visible, no reset is written, and the template's 0 survives — which is +// exactly what the reporter kept seeing on a binary that contained the fix. +// +// The defect is which value the condition is evaluated against. `maxHeightUnit` +// is unmapped, so it is written with whatever the TEMPLATE captured — "none" — +// and never with its declared "pixels". Asking "is maxHeight hidden?" of the +// declared default asks about a document that will not be written; it has to be +// asked of the configuration that WILL be, which is what `stored` carries. +// +// Ground truth for the answer, measured across the 69 Image widgets in that +// project: all 65 carrying a `maxHeight` store **250**, at every combination of +// heightUnit and maxHeightUnit. The only outlier was the one mxcli authored. +// Proven both ways on the real package — patching that stored 0 to 250 takes the +// project from 1 error to 0, and restoring it takes it back to 1. // imageDefWithMaxHeight is the Image widget reduced to the two properties this is // about — one hidden property WITH an MDL mapping, one WITHOUT. @@ -48,20 +74,37 @@ func imageDefWithMaxHeight() *WidgetDefinition { return def } +// The DECLARED defaults, as Image 1.6.0 states them — `maxHeightUnit` is +// "pixels". Reading "none" here is what let the first fix pass a test it could +// not pass in a project. func imageDefaultsWithMaxHeight() map[string]string { d := imageDefaults() d["maxheight"] = "250" - d["maxheightunit"] = "none" + d["maxheightunit"] = "pixels" return d } +// The TEMPLATE's captured configuration, as mxcli's embedded Image template +// holds it. `maxHeightUnit` is "none", which is NOT its declared default — a +// template captures whatever the widget it was extracted from happened to be set +// to, and that is precisely why an unmapped property cannot be reasoned about +// from the declared defaults. +func imageTemplateValues() map[string]string { + return map[string]string{ + "heightUnit": "auto", "widthUnit": "auto", + "maxHeightUnit": "none", "minHeightUnit": "none", + "maxHeight": "0", "minHeight": "0", + } +} + // The reset list must include a hidden property that has no MDL mapping. Before // this, the caller iterated mappings, so `maxHeight` could be named here and // still never written. func TestHiddenResets_IncludeAPropertyMDLCannotName(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} def := imageDefWithMaxHeight() - hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, imageDefaultsWithMaxHeight()) + hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, + imageDefaultsWithMaxHeight(), imageTemplateValues()) resets := unmappedHiddenResets(def, def.PropertyVisibility, hidden) got, ok := resets["maxHeight"] @@ -73,13 +116,33 @@ func TestHiddenResets_IncludeAPropertyMDLCannotName(t *testing.T) { } } +// THE CONTROL that makes the test above mean anything, and the one the first +// attempt at this fix did not have. Drop the template's values and the rule is +// evaluated against the DECLARED default "pixels" instead — the condition is +// false, maxHeight reads as visible, and nothing is written. That is the exact +// state of the shipped binary the reporter measured: the fix present, the value +// still 0. +func TestHiddenResets_DeclaredDefaultAloneCannotAnswerTheCondition(t *testing.T) { + e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} + def := imageDefWithMaxHeight() + hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, + imageDefaultsWithMaxHeight(), nil) + + if v, ok := unmappedHiddenResets(def, def.PropertyVisibility, hidden)["maxHeight"]; ok { + t.Errorf("maxHeight was reset to %q from the declared defaults alone — if this "+ + "passes, the condition no longer depends on the template's value and the "+ + "test above proves nothing", v) + } +} + // CONTROL 1: a hidden property that DOES have a mapping must not appear here. // The mapping loop already writes it, and writing it twice would be a second // value for one property. func TestHiddenResets_SkipWhatTheMappingLoopAlreadyWrites(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} def := imageDefWithMaxHeight() - hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, imageDefaultsWithMaxHeight()) + hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, + imageDefaultsWithMaxHeight(), imageTemplateValues()) for _, mapped := range []string{"width", "height"} { if _, ok := unmappedHiddenResets(def, def.PropertyVisibility, hidden)[mapped]; ok { @@ -95,8 +158,10 @@ func TestHiddenResets_SkipWhatTheMappingLoopAlreadyWrites(t *testing.T) { func TestHiddenResets_NoDeclaredDefaultWritesNothing(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} def := imageDefWithMaxHeight() - // Defaults for everything except maxHeight. - hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, imageDefaults()) + // The template still says maxHeightUnit is "none", so the property IS + // hidden; what is missing is anything to reset it TO. + hidden := e.hiddenUnnamedProperties(def, &ast.WidgetV3{Name: "img"}, + imageDefaults(), imageTemplateValues()) if v, ok := unmappedHiddenResets(def, def.PropertyVisibility, hidden)["maxHeight"]; ok { t.Errorf("maxHeight was reset to %q with no declared default to reset it to", v) @@ -106,13 +171,16 @@ func TestHiddenResets_NoDeclaredDefaultWritesNothing(t *testing.T) { // CONTROL 3: a VISIBLE unmapped property is not touched. The reset is about // hidden properties; applying it to a visible one would overwrite the template's // value where the template is right. +// +// It also pins the precedence: the script's own `MaxHeightUnit: pixels` outranks +// the template's captured "none", or naming a property would stop meaning +// anything. func TestHiddenResets_VisibleUnmappedPropertyIsLeftAlone(t *testing.T) { e := &PluggableWidgetEngine{pageBuilder: &pageBuilder{}} def := imageDefWithMaxHeight() - // maxHeightUnit = "pixels" makes maxHeight visible, so nothing is hidden by - // that rule. w := &ast.WidgetV3{Name: "img", Properties: map[string]any{"MaxHeightUnit": "pixels"}} - hidden := e.hiddenUnnamedProperties(def, w, imageDefaultsWithMaxHeight()) + hidden := e.hiddenUnnamedProperties(def, w, + imageDefaultsWithMaxHeight(), imageTemplateValues()) if _, ok := unmappedHiddenResets(def, def.PropertyVisibility, hidden)["maxHeight"]; ok { t.Error("maxHeight is visible under this configuration and must keep its value") diff --git a/mdl/executor/widget_item_action_slot_test.go b/mdl/executor/widget_item_action_slot_test.go index 2e15ca4a9e..96078d932d 100644 --- a/mdl/executor/widget_item_action_slot_test.go +++ b/mdl/executor/widget_item_action_slot_test.go @@ -191,7 +191,7 @@ func TestExtractObjectListItem_ReadsAnAction(t *testing.T) { var got string for _, p := range item.Props { - if p.Key == "StaticOnClickAction" { + if p.Key == "staticOnClickAction" { if !p.IsRef { t.Error("the action was quoted — `staticOnClickAction: 'microflow …'` does not parse back") } diff --git a/mdl/executor/widget_item_template_params_test.go b/mdl/executor/widget_item_template_params_test.go index b6c257565c..af3fad1319 100644 --- a/mdl/executor/widget_item_template_params_test.go +++ b/mdl/executor/widget_item_template_params_test.go @@ -42,12 +42,12 @@ func TestExtractObjectListItem_EmitsTextTemplateParameters(t *testing.T) { var text, params string for _, p := range item.Props { switch p.Key { - case "ButtonCaption": + case "buttonCaption": text = p.Value - case "ButtonCaptionParams": + case "buttonCaptionParams": params = p.Value if !p.IsRef { - t.Error("the parameter list was quoted — `ButtonCaptionParams: '[...]'` does not parse") + t.Error("the parameter list was quoted — `buttonCaptionParams: '[...]'` does not parse") } } } @@ -78,7 +78,7 @@ func TestBuildObjectListItem_ReadsParamsCompanionCaseInsensitively(t *testing.T) }, } child := &ast.WidgetV3{Name: "b1", Properties: map[string]any{ - "ButtonCaption": "Hello {1}", + "buttonCaption": "Hello {1}", "ButtonCaptionParams": []ast.ParamAssignmentV3{{Index: 1, Value: "'abc'"}}, }} diff --git a/mdl/executor/widget_known_props_from_mpk.go b/mdl/executor/widget_known_props_from_mpk.go new file mode 100644 index 0000000000..2f0e38cfe7 --- /dev/null +++ b/mdl/executor/widget_known_props_from_mpk.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "path/filepath" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// enrichKnownPropertiesFromMPK fills each definition's KnownProperties from the +// widget's INSTALLED package, so the property validator recognises every +// property the widget actually declares. +// +// # Why +// +// mxcli reads a widget from two places. DESCRIBE WIDGET parses the project's +// .mpk (version-accurate, and the only place a Marketplace widget appears). The +// property validator reads the WidgetDefinition. For most widgets those agree, +// because the .def.json cache is GENERATED from the .mpk and its +// knownProperties already carry everything unmapped. +// +// Nine widgets are the exception. COMBOBOX, GALLERY, IMAGE, BARCODESCANNER, the +// four data-grid filters and DROPDOWNSORT have hand-crafted definitions in +// sdk/widgets/definitions/ and are deliberately never extracted per-project, so +// their .def.json never exists and their hand-written property list is whatever +// someone typed. Measured against the packages in testdata/expr-checker: +// +// combobox 73 properties in the .mpk, 7 mapped + 4 known +// gallery 44 12 +// image 37 14 +// barcodescanner 12 1 +// +// So DESCRIBE emitted an example it labels "parses as written" — and it does +// parse — naming properties the validator then rejected as nonexistent. Because +// exec refuses a script with errors, BARCODESCANNER could not be placed from MDL +// in any legal form: name the five properties and exec refuses, omit them and +// mxbuild reports CE0463 "the definition of this widget has changed". +// +// # Known, not allowed +// +// A .mpk property with no mapping is added to KnownProperties, NOT to the +// allowed set. That is the honest distinction the validator already draws: +// knownUnmappedProperties turns it into MDL-WIDGET06 — "recognized but not yet +// persisted by mxcli; a non-default value will be dropped" — rather than +// silently accepting a value nothing writes. Promoting them to "allowed" would +// trade a false error for a silent drop. +// +// MDL-WIDGET01 keeps its job: a name no package declares is still an error, so +// typos are still caught. +// +// # Applied to every definition, not to a list of nine +// +// Recomputing KnownProperties for a generated definition produces what +// generation already put there, so the enrichment is idempotent where it is +// redundant. Naming the nine would be the same hand-maintained-list defect one +// layer up — the defect this whole line of work exists to remove. +func enrichKnownPropertiesFromMPK(r *WidgetRegistry, projectPath string) { + if r == nil || projectPath == "" { + return + } + byID := mpkPropertiesByWidgetID(filepath.Dir(projectPath)) + if len(byID) == 0 { + return + } + for _, def := range r.byWidgetID { + props, ok := byID[def.WidgetID] + if !ok { + continue + } + allowed, _ := allowedWidgetProperties(def) + seen := make(map[string]bool, len(def.KnownProperties)) + for _, k := range def.KnownProperties { + seen[strings.ToLower(k)] = true + } + var added []string + for _, p := range props { + if p.Key == "" || p.IsSystem { + continue + } + l := strings.ToLower(p.Key) + if allowed[l] || seen[l] { + continue + } + seen[l] = true + added = append(added, p.Key) + } + if len(added) == 0 { + continue + } + sort.Strings(added) + def.KnownProperties = append(def.KnownProperties, added...) + } +} + +// mpkPropertiesByWidgetID parses every package in the project's widgets/ folder +// once and indexes the properties by widget id. +// +// ParseAll rather than ParseMPK: a bundled package (Charts.mpk) carries many +// widgets and ParseMPK returns only the first, which is the #679 bug — here it +// would silently leave every chart but one unenriched. +func mpkPropertiesByWidgetID(projectDir string) map[string][]mpk.PropertyDef { + matches, err := filepath.Glob(filepath.Join(projectDir, "widgets", "*.mpk")) + if err != nil || len(matches) == 0 { + return nil + } + out := make(map[string][]mpk.PropertyDef, len(matches)) + for _, path := range matches { + defs, err := mpk.ParseAll(path) + if err != nil { + continue // a package we cannot read enriches nothing; it is not an error + } + for _, d := range defs { + if d == nil || d.ID == "" { + continue + } + out[d.ID] = d.Properties + } + } + return out +} diff --git a/mdl/executor/widget_primitive_default_condition_test.go b/mdl/executor/widget_primitive_default_condition_test.go index e251cc0783..c9b9cfc918 100644 --- a/mdl/executor/widget_primitive_default_condition_test.go +++ b/mdl/executor/widget_primitive_default_condition_test.go @@ -67,7 +67,7 @@ func TestHiddenUnnamedProperties_PrunesTheInactiveDataSource(t *testing.T) { def := uploadModeDef() filesMode := e.hiddenUnnamedProperties(def, &ast.WidgetV3{ - Name: "fu", Properties: map[string]any{"DataSource": "assoc"}}, nil) + Name: "fu", Properties: map[string]any{"DataSource": "assoc"}}, nil, nil) if _, ok := filesMode["associatedimages"]; !ok { t.Error("associatedImages was not pruned under the default uploadMode — this is the CE0463") } @@ -76,7 +76,7 @@ func TestHiddenUnnamedProperties_PrunesTheInactiveDataSource(t *testing.T) { } imagesMode := e.hiddenUnnamedProperties(def, &ast.WidgetV3{ - Name: "fu", Properties: map[string]any{"uploadMode": "images", "DataSource": "assoc"}}, nil) + Name: "fu", Properties: map[string]any{"uploadMode": "images", "DataSource": "assoc"}}, nil, nil) if _, ok := imagesMode["associatedfiles"]; !ok { t.Error("associatedFiles was not pruned under uploadMode images") } diff --git a/mdl/exprcheck/adapters/check.go b/mdl/exprcheck/adapters/check.go index 7623bdd821..13410cdbcd 100644 --- a/mdl/exprcheck/adapters/check.go +++ b/mdl/exprcheck/adapters/check.go @@ -124,6 +124,11 @@ func (c *CheckAdapter) walkBodyWithScope(body []ast.MicroflowStatement, mf strin c.checkExpr(n.Value, "MfSetStmt.Value", mf, r) case *ast.LogStmt: c.checkExpr(n.Message, "LogStmt.Message", mf, r) + // The template parameters were not walked at all, so a non-String + // one reached mxbuild as CE0117 (mendixlabs/mxcli#1043). + for _, tp := range n.Template { + c.checkExpr(tp.Value, "LogStmt.TemplateParam", mf, r) + } case *ast.CreateObjectStmt: entityQN := n.EntityType.String() for _, ci := range n.Changes { diff --git a/mdl/exprcheck/ast.go b/mdl/exprcheck/ast.go index 04581b3995..f13a7be041 100644 --- a/mdl/exprcheck/ast.go +++ b/mdl/exprcheck/ast.go @@ -38,6 +38,11 @@ type EmptyExpr struct{ baseNode } type VariableExpr struct { baseNode Name string + // Bare is true when the source wrote a plain identifier (`Closed`) rather + // than a variable reference (`$Closed`). Mendix has no bare identifiers in + // expressions, so the distinction is a defect signal — see + // checkBareIdentifierValue. + Bare bool } type AttributePathExpr struct { diff --git a/mdl/exprcheck/hints/registry.go b/mdl/exprcheck/hints/registry.go index 734e4ba2aa..9f4ff6ece9 100644 --- a/mdl/exprcheck/hints/registry.go +++ b/mdl/exprcheck/hints/registry.go @@ -175,6 +175,46 @@ var Registry = ®istry{byCode: map[string]Entry{ }, }, }, + "E014": { + Code: "E014", + Slug: "trailing-tokens", + Severity: SeverityError, + Trigger: "The expression parsed, but tokens were left over after it.", + WhyWrong: "The expression is incomplete or malformed — most often a Mendix keyword used as if it were a function, or two keywords glued together.", + HowToFix: "`empty` is a keyword, not a function: write `$List = empty` rather than `empty($List)`. For a count, use `length($List) = 0`.", + Examples: []ExampleFix{ + { + Wrong: "IF empty($Orders) THEN", + Right: "IF $Orders = empty THEN", + Note: "empty is a keyword, so the '(' is left over", + }, + { + Wrong: "IF $X = '' emptyor $Y THEN", + Right: "IF $X = '' empty or $Y THEN", + Note: "glued keywords", + }, + }, + }, + "E013": { + Code: "E013", + Slug: "bare-identifier-value", + Severity: SeverityError, + Trigger: "A bare word stands alone as the value of a create/change member.", + WhyWrong: "Mendix expressions have no bare identifiers: a value is a literal, a $variable, a qualified name or a function call. The bare word reaches the build as CE0117 \"Error(s) in expression\".", + HowToFix: "Quote it if it is text, add the $ if it is a variable, or qualify it if it is an enumeration value.", + Examples: []ExampleFix{ + { + Wrong: "CHANGE $Order (Status = Closed);", + Right: "CHANGE $Order (Status = 'Closed');", + Note: "a String attribute takes a quoted literal", + }, + { + Wrong: "CHANGE $Order (Status = Closed);", + Right: "CHANGE $Order (Status = Sales.OrderStatus.Closed);", + Note: "an enumeration attribute takes a qualified value", + }, + }, + }, "E012": { Code: "E012", Slug: "id-attribute-illegal", diff --git a/mdl/exprcheck/parser.go b/mdl/exprcheck/parser.go index 89aa26330c..2c69f0df9e 100644 --- a/mdl/exprcheck/parser.go +++ b/mdl/exprcheck/parser.go @@ -32,20 +32,132 @@ func (p *parserImpl) Parse(src string, ctx Context) (RobustExpr, []Hint) { // would produce false positives for valid expressions that use characters // the lexer does not model (e.g. "$Total : $Count" with Mendix ':' division). if t := s.Peek(); t.Kind != TokEOF && t.Kind != TokError { + // This hint used to carry no code, no document and no microflow: the + // location held only a line and column, which are the offsets WITHIN the + // expression fragment rather than into the file, so they printed as + // nothing useful. The reader saw `[]` for the code and had to find the + // offending expression in a 40-line script by eye + // (mendixlabs/mxcli#1042). Everything the context knows is attached now, + // which is what hintsLocation is for. + // + // The fix line is also no longer only about glued keywords. That was the + // case it was written for, and it sent people hunting for an 'emptyor' + // that was not there: the report's actual input was `empty($List)` — + // `empty` is a Mendix KEYWORD, not a function, so the parser consumed it + // and stopped at the '('. Naming the likelier causes first makes the + // message point at the real one. hs = append(hs, hints.Hint{ + Code: "E014", + Slug: "trailing-tokens", Severity: hints.SeverityError, - Where: hints.Location{ - Line: t.Pos.Line, - Column: t.Pos.Column, - }, + Where: hintsLocation(ctx, t.Pos), YouWrote: t.Text, - Problem: "Unexpected token after expression — the expression appears incomplete or malformed (possible missing space between keywords).", - Fix: "Check for glued keywords such as 'emptyor' (should be 'empty or') or 'andtrue' (should be 'and true').", + Problem: "Unexpected token after the expression — everything up to here parsed, " + + "and this is left over, so the expression is incomplete or malformed.", + Fix: "Check that a keyword is not being used as a function — `empty` is a keyword, " + + "so `empty($List)` is not a call; write `$List = empty` (or `length($List) = 0`). " + + "Also check for glued keywords such as 'emptyor' (should be 'empty or').", }) } + hs = append(hs, checkSlotKind(expr, ctx)...) + hs = append(hs, checkBareIdentifierValue(expr, ctx)...) return expr, hs } +// checkSlotKind compares the whole expression's inferred kind against what the +// slot it sits in expects. +// +// The expectations table in slot_resolver.go had existed for some time with +// NOTHING READING IT — slotKind() was defined and never called — so a slot +// declared `{Kind: KindString}` constrained nothing. `LOG WARNING 42` passed, +// and so did every non-String log template parameter (mendixlabs/mxcli#1043), +// which is CE0117 "Error(s) in expression" at the activity. +// +// Only a CONCRETE expectation is enforced. An entry carrying ResolveBy +// ("AttributeOf:Parent", "MicroflowReturn", …) names a kind that has to be +// looked up per call site, and the adapter encodes that by appending the +// resolved target to the slot path ("ChangeItem.Value:Sales.Order.Status"), +// which does not match the table at all — so those stay unenforced here rather +// than being enforced against the wrong kind. +// +// Both sides must be known. An inferred KindUnknown is "could not tell", and +// reporting it would flag every expression whose type mxcli cannot resolve. +func checkSlotKind(expr RobustExpr, ctx Context) []Hint { + sc, ok := slotKind(ctx) + if !ok || sc.Kind == KindUnknown || sc.ResolveBy != "" { + return nil + } + k := inferKind(expr, ctx) + if k == KindUnknown || k == sc.Kind { + return nil + } + // `empty` satisfies any slot: it is Mendix's null, not a kind of its own. + if k == KindEmpty { + return nil + } + fix := "Convert it, e.g. with toString(...)." + if sc.Kind != KindString { + fix = "Replace it with an expression of kind " + typeKindName(sc.Kind) + "." + } + return []Hint{{ + Code: "E009", + Slug: "slot-type-mismatch", + Severity: hints.SeverityError, + Where: hintsLocation(ctx, expr.Pos()), + YouWrote: "<" + typeKindName(k) + ">", + Problem: "This position requires " + typeKindName(sc.Kind) + ", but the expression has kind " + + typeKindName(k) + ". Mendix does not coerce here — it reports CE0117 \"Error(s) in expression\".", + Fix: fix, + }} +} + +// checkBareIdentifierValue reports a bare word standing alone as a member's +// value: `CHANGE $Order (Status = Closed)`. +// +// Mendix expressions have no bare identifiers — a variable is `$Name`, a string +// is 'quoted', an enumeration value is Module.Enum.Value — so the parser reads +// one as a variable reference, it resolves to nothing, and the kind comes out +// Unknown. Unknown is tolerated everywhere by design, which is exactly why this +// slipped through check and exec to arrive as CE0117 (mendixlabs/mxcli#1044). +// +// Scoped to the WHOLE expression of a create/change member, and that scoping is +// the load-bearing part rather than caution for its own sake: a bare name NESTED +// inside a list-operation predicate is legal MDL — `FILTER($L, Status = 'Open')` +// resolves `Status` against the item under test — so a rule that fired on any +// bare identifier would reject working scripts. A member's value is the one +// position where the bare word is the entire expression and can only be a +// mistake. +func checkBareIdentifierValue(expr RobustExpr, ctx Context) []Hint { + if !strings.HasPrefix(ctx.SlotPath, "ChangeItem.Value") && + !strings.HasPrefix(ctx.SlotPath, "CreateItem.Value") { + return nil + } + v, ok := expr.(*VariableExpr) + if !ok || !v.Bare { + return nil + } + // A name that IS in scope is a variable the author spelled without its $, + // which is worth saying differently. + fix := "Quote it if it is text ('" + v.Name + "'), write $" + v.Name + + " if it is a variable, or qualify it (Module.Enum.Value) if it is an enumeration value." + if ctx.Scope != nil { + if _, inScope := ctx.Scope.Lookup(v.Name); inScope { + fix = "Write $" + v.Name + " — a variable reference needs its $." + } + } + return []Hint{{ + Code: "E013", + Slug: "bare-identifier-value", + Severity: hints.SeverityError, + Where: hintsLocation(ctx, expr.Pos()), + YouWrote: v.Name, + Problem: "A bare word is not a Mendix expression. Mendix reads a value as a literal, " + + "a $variable, a qualified name or a function call, so this arrives as " + + "CE0117 \"Error(s) in expression\".", + Fix: fix, + }} +} + func parseOr(s *Stream, ctx Context) (RobustExpr, []Hint) { left, hints := parseAnd(s, ctx) first := true @@ -389,7 +501,10 @@ func parseIdentLed(s *Stream, ctx Context) (RobustExpr, []Hint) { } return &QNameExpr{baseNode: baseNode{P: t.Pos}, Module: name, Name: n2}, nil } - return &VariableExpr{baseNode: baseNode{P: t.Pos}, Name: name}, nil + // Bare records that this came from a plain identifier rather than + // `$Name`. The two collapse to the same node, and telling them apart is + // what makes the bare-word rule possible. + return &VariableExpr{baseNode: baseNode{P: t.Pos}, Name: name, Bare: true}, nil } // parseQualifiedCall consumes the argument list of a `Module.Name(...)` call. @@ -570,6 +685,15 @@ func inferKind(e RobustExpr, ctx Context) TypeKind { return k } } + // A variable the entity scope knows holds an OBJECT. Without this a + // `$Customer` infers Unknown, and Unknown is tolerated everywhere — so + // an object handed to a slot that wants a String went unreported, which + // is the case mendixlabs/mxcli#1043 was actually filed about. + if ctx.Entities != nil { + if _, ok := ctx.Entities.VariableEntity(n.Name); ok { + return KindObject + } + } case *CallExpr: if sig, ok := funcTable[n.Name]; ok { if sig.retFromArgs { diff --git a/mdl/exprcheck/slot_kind_test.go b/mdl/exprcheck/slot_kind_test.go new file mode 100644 index 0000000000..43a66615ab --- /dev/null +++ b/mdl/exprcheck/slot_kind_test.go @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: Apache-2.0 + +package exprcheck + +import ( + "strings" + "testing" +) + +// The slot-expectation table had existed with NOTHING READING IT — slotKind() +// was defined and never called — so a slot declared `{Kind: KindString}` +// constrained nothing. `LOG WARNING 42` passed, and so did every non-String log +// template parameter, which mxbuild reports as CE0117 "Error(s) in expression" +// (mendixlabs/mxcli#1043). + +// fakeEntities answers the object half of inference: a variable the entity +// scope knows holds an object. +type fakeEntities map[string]string + +func (f fakeEntities) VariableEntity(name string) (string, bool) { + qn, ok := f[strings.TrimPrefix(name, "$")] + return qn, ok +} +func (fakeEntities) AssociationTarget(string, string) (string, bool) { return "", false } + +func parseIn(t *testing.T, src, slot string, opts ...func(*Context)) []Hint { + t.Helper() + ctx := Context{SlotPath: slot, Slots: DefaultSlotResolver()} + for _, o := range opts { + o(&ctx) + } + _, hs := NewParser().Parse(src, ctx) + return hs +} + +func codes(hs []Hint) []string { + var out []string + for _, h := range hs { + out = append(out, h.Code) + } + return out +} + +// Measured on 11.13.0 by executing and building each: Boolean, DateTime, +// Decimal and Integer log parameters all fail CE0117; a String one is clean; +// toString(...) around any of them is clean. +func TestSlotKind_LogTemplateParamMustBeString(t *testing.T) { + for _, src := range []string{"42", "true", "1.5"} { + hs := parseIn(t, src, "LogStmt.TemplateParam") + if !hasCode(hs, "E009") { + t.Errorf("%q in a String slot was accepted: %v", src, codes(hs)) + } + } + + // CONTROL: a String is fine, and so is the conversion mxbuild accepts. + for _, src := range []string{"'text'", "toString(42)"} { + if hs := parseIn(t, src, "LogStmt.TemplateParam"); hasCode(hs, "E009") { + t.Errorf("%q was rejected in a String slot: %v", src, hs) + } + } +} + +// The case #1043 was actually filed about. A variable holding an OBJECT infers +// Unknown without the entity scope, and Unknown is tolerated everywhere — so +// the object went unreported. +func TestSlotKind_AnObjectIsNotAString(t *testing.T) { + withEnts := func(c *Context) { c.Entities = fakeEntities{"Customer": "Sales.Customer"} } + + hs := parseIn(t, "$Customer", "LogStmt.TemplateParam", withEnts) + if !hasCode(hs, "E009") { + t.Errorf("an object in a String slot was accepted: %v", codes(hs)) + } + + // CONTROL: without the entity scope the kind is genuinely unknown, and + // unknown must stay silent rather than be guessed at. + if hs := parseIn(t, "$Customer", "LogStmt.TemplateParam"); hasCode(hs, "E009") { + t.Errorf("reported a variable whose kind could not be established: %v", hs) + } +} + +// CONTROL for the scope of the whole mechanism. A slot whose expectation has to +// be resolved per call site (ResolveBy) is NOT enforced here: the adapter +// encodes the resolved target by appending it to the slot path, so the table +// entry does not apply and enforcing it would compare against the wrong kind. +func TestSlotKind_ResolveBySlotsAreNotEnforced(t *testing.T) { + for _, slot := range []string{ + "ChangeItem.Value", + "ChangeItem.Value:Sales.Order.Qty", + "ReturnStmt.Value", + "CallArgument.Value", + } { + if hs := parseIn(t, "42", slot); hasCode(hs, "E009") { + t.Errorf("slot %q was enforced against its placeholder kind: %v", slot, hs) + } + } +} + +// `empty` is Mendix's null rather than a kind of its own, so it satisfies any +// slot. Without this every `LOG … WITH ({1} = empty)` would be reported. +func TestSlotKind_EmptySatisfiesAnySlot(t *testing.T) { + if hs := parseIn(t, "empty", "LogStmt.TemplateParam"); hasCode(hs, "E009") { + t.Errorf("empty was rejected in a String slot: %v", hs) + } +} + +// CONTROL: a slot with no table entry constrains nothing. +func TestSlotKind_UnknownSlotIsSilent(t *testing.T) { + if hs := parseIn(t, "42", "NoSuchSlot.Value"); hasCode(hs, "E009") { + t.Errorf("an unmapped slot was enforced: %v", hs) + } +} + +// --------------------------------------------------------------------------- +// E013 — a bare word as a member's value (mendixlabs/mxcli#1044) +// --------------------------------------------------------------------------- + +// Mendix expressions have no bare identifiers, so the parser reads one as a +// variable, it resolves to nothing, and the kind comes out Unknown — which is +// tolerated everywhere. That is why `CHANGE $Order (Status = Closed)` passed +// check and exec and arrived as CE0117. +func TestBareIdentifier_ReportedAsAMemberValue(t *testing.T) { + for _, slot := range []string{ + "ChangeItem.Value:Sales.Order.Status", + "CreateItem.Value:Sales.Order.Status", + } { + hs := parseIn(t, "Closed", slot) + if !hasCode(hs, "E013") { + t.Errorf("slot %q accepted a bare word: %v", slot, codes(hs)) + } + for _, h := range hs { + if h.Code == "E013" && !strings.Contains(h.Fix, "'Closed'") { + t.Errorf("the fix should offer the quoted form: %s", h.Fix) + } + } + } +} + +// THE CONTROL that scopes the rule. A bare name NESTED in a list-operation +// predicate is legal MDL — `FILTER($L, Status = 'Open')` resolves `Status` +// against the item under test — so a rule that fired on any bare identifier +// would reject working scripts. Only a bare word that is the WHOLE expression +// of a member value can only be a mistake. +func TestBareIdentifier_NotReportedWhenNested(t *testing.T) { + for _, tc := range []struct{ src, slot string }{ + {"FILTER($L, Status = 'Open')", "MfSetStmt.Value"}, + {"FIND($L, Code = 'x')", "ChangeItem.Value:Sales.Order.Ref"}, + {"Status = 'Open'", "ChangeItem.Value:Sales.Order.Flag"}, + } { + if hs := parseIn(t, tc.src, tc.slot); hasCode(hs, "E013") { + t.Errorf("%q reported a nested bare name: %v", tc.src, hs) + } + } +} + +// CONTROL: outside a member value the rule is silent, because a bare word may +// be legitimate there and this rule has only been established for one position. +func TestBareIdentifier_OnlyInMemberValues(t *testing.T) { + for _, slot := range []string{"IfStmt.Condition", "LogStmt.Message", "MfSetStmt.Value"} { + if hs := parseIn(t, "Closed", slot); hasCode(hs, "E013") { + t.Errorf("slot %q was reported: %v", slot, hs) + } + } +} + +// CONTROL: every spelling that IS a Mendix expression must pass. +func TestBareIdentifier_AcceptsRealExpressions(t *testing.T) { + for _, src := range []string{ + "'Closed'", // a string literal + "$Closed", // a variable + "Sales.OrderStatus.Closed", // a qualified enumeration value + "toString($Order/Qty)", // a call + "true", // a keyword literal + "empty", // Mendix's null + "[%CurrentDateTime%]", // a token + } { + if hs := parseIn(t, src, "ChangeItem.Value:Sales.Order.Status"); hasCode(hs, "E013") { + t.Errorf("%q was reported as a bare word: %v", src, hs) + } + } +} + +// The trailing-token hint carried no code and no document: the reader saw `[]` +// where a code belongs and had to find the offending expression in a 40-line +// script by eye. Its fix line also named only glued keywords, which sent people +// hunting for an 'emptyor' that was not there — the reported input was +// `empty($List)`, where `empty` is a Mendix KEYWORD rather than a function, so +// the parser consumed it and stopped at the '(' (mendixlabs/mxcli#1042). +func TestTrailingTokens_CarriesACodeAndTheRealCause(t *testing.T) { + hs := parseIn(t, "empty($Orders)", "IfStmt.Condition", func(c *Context) { + c.Microflow = "Bench.P_Empty" + }) + var got *Hint + for i := range hs { + if hs[i].Code == "E014" { + got = &hs[i] + } + } + if got == nil { + t.Fatalf("no E014 for a trailing token: %v", codes(hs)) + } + // Which microflow it is in, since the line/column are offsets into the + // expression fragment rather than into the file. + if got.Where.Microflow != "Bench.P_Empty" { + t.Errorf("the hint does not say which flow it is in: %+v", got.Where) + } + // The likelier cause first — `empty` as a function, not a glued keyword. + if !strings.Contains(got.Fix, "keyword") || !strings.Contains(got.Fix, "$List = empty") { + t.Errorf("the fix should name the keyword-as-function cause: %s", got.Fix) + } +} + +// CONTROL: the correct spellings parse with nothing left over. +func TestTrailingTokens_AcceptsTheCorrectSpellings(t *testing.T) { + for _, src := range []string{ + "$Orders = empty", + "length($Orders) = 0", + "$X = '' or $Y = ''", + } { + if hs := parseIn(t, src, "IfStmt.Condition"); hasCode(hs, "E014") { + t.Errorf("%q reported a trailing token: %v", src, hs) + } + } +} diff --git a/mdl/exprcheck/slot_resolver.go b/mdl/exprcheck/slot_resolver.go index af28958183..7a243e489d 100644 --- a/mdl/exprcheck/slot_resolver.go +++ b/mdl/exprcheck/slot_resolver.go @@ -6,15 +6,21 @@ package exprcheck // Add a new entry whenever a new MDL statement slot is added to the executor. // Slot paths mirror the AST node + field name, e.g. "IfStmt.Condition". var staticExpectations = map[string]SlotConstraint{ - "IfStmt.Condition": {Kind: KindBoolean}, - "WhileStmt.Condition": {Kind: KindBoolean}, - "RetrieveStmt.LimitExpr": {Kind: KindInteger}, - "RetrieveStmt.OffsetExpr": {Kind: KindInteger}, - "ChangeItem.Value": {Kind: KindUnknown, ResolveBy: "AttributeOf:Parent"}, - "CreateItem.Value": {Kind: KindUnknown, ResolveBy: "AttributeOf:Parent"}, - "ReturnStmt.Value": {Kind: KindUnknown, ResolveBy: "MicroflowReturn"}, - "CallArgument.Value": {Kind: KindUnknown, ResolveBy: "TargetParameter"}, - "LogStmt.Message": {Kind: KindString}, + "IfStmt.Condition": {Kind: KindBoolean}, + "WhileStmt.Condition": {Kind: KindBoolean}, + "RetrieveStmt.LimitExpr": {Kind: KindInteger}, + "RetrieveStmt.OffsetExpr": {Kind: KindInteger}, + "ChangeItem.Value": {Kind: KindUnknown, ResolveBy: "AttributeOf:Parent"}, + "CreateItem.Value": {Kind: KindUnknown, ResolveBy: "AttributeOf:Parent"}, + "ReturnStmt.Value": {Kind: KindUnknown, ResolveBy: "MicroflowReturn"}, + "CallArgument.Value": {Kind: KindUnknown, ResolveBy: "TargetParameter"}, + "LogStmt.Message": {Kind: KindString}, + // A log message's template parameter must be a String — Mendix does not + // coerce here, and a non-String one is CE0117 "Error(s) in expression" on + // the activity. Measured on 11.13.0: an Integer attribute, an integer + // literal and an object each fail; a String attribute is clean; and + // toString(...) around any of them is clean (mendixlabs/mxcli#1043). + "LogStmt.TemplateParam": {Kind: KindString}, "MfSetStmt.Value": {Kind: KindUnknown, ResolveBy: "TargetVariable"}, "DeclareStmt.InitialValue": {Kind: KindUnknown, ResolveBy: "DeclareType"}, } diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index f40d3b7f44..40896bbff6 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -383,6 +383,9 @@ READONLY: R E A D O N L Y; ATTRIBUTES: A T T R I B U T E S; FILTERTYPE: F I L T E R T Y P E; IMAGE: I M A G E; +// GLYPH names Mendix's legacy icon element (Forms$GlyphIcon), a numeric +// character code rather than a reference into a collection. +GLYPH: G L Y P H; QUEUE: Q U E U E; QUEUES: Q U E U E S; SCHEDULED: S C H E D U L E D; diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index 2725087072..a78fc1534f 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -342,8 +342,30 @@ navigationClause // which is why it sits beside PAGE and MICROFLOW rather than in a syntax of its // own. navMenuItemDef - : MENU_KW ITEM STRING_LITERAL ((PAGE qualifiedName) | (MICROFLOW qualifiedName) | SIGN_OUT)? (ICON qualifiedName)? SEMICOLON? - | MENU_KW STRING_LITERAL (ICON qualifiedName)? LPAREN navMenuItemDef* RPAREN SEMICOLON? + : MENU_KW ITEM STRING_LITERAL ((PAGE qualifiedName) | (MICROFLOW qualifiedName) | SIGN_OUT)? navMenuIcon? SEMICOLON? + | MENU_KW STRING_LITERAL navMenuIcon? LPAREN navMenuItemDef* RPAREN SEMICOLON? + ; + +// Mendix stores three DIFFERENT icon elements, and they are not variants of one +// value: an icon-collection icon and an image icon each hold a qualified name — +// into an icon collection and an image collection, which are different documents +// — while a glyph icon holds a numeric character code and no name at all. +// +// ICON Atlas_Core.Atlas.home Forms$IconCollectionIcon +// ICON GLYPH 57345 Forms$GlyphIcon +// ICON IMAGE MyModule.Images.logo Forms$ImageIcon +// +// Only the first was expressible, so DESCRIBE emitted a comment for the other +// two and re-running its own output DESTROYED them. +// +// The two keyword-led alternatives come FIRST. qualifiedName accepts a keyword +// as a name segment (identifierOrKeyword), so `ICON IMAGE …` also matches the +// bare form with `image` read as the name; listing the specific alternatives +// ahead of the general one is what settles it. +navMenuIcon + : ICON GLYPH NUMBER_LITERAL + | ICON IMAGE qualifiedName + | ICON qualifiedName ; // A standalone menu document (Menus$MenuDocument) — the reusable menu a menu diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index 5eb25b5225..02b1b5b3df 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -172,6 +172,8 @@ describeStatement | DESCRIBE ODATA SERVICE qualifiedName // DESCRIBE ODATA SERVICE Module.ServiceName | DESCRIBE EXTERNAL ENTITY qualifiedName // DESCRIBE EXTERNAL ENTITY Module.EntityName | DESCRIBE NAVIGATION (qualifiedName | IDENTIFIER)? // DESCRIBE NAVIGATION [profile] + | DESCRIBE WIDGET identifierOrKeyword // DESCRIBE WIDGET combobox | DESCRIBE WIDGET 'com.mendix…' + | DESCRIBE WIDGET STRING_LITERAL // …by full widget id, which contains dots | DESCRIBE STYLING ON (PAGE | SNIPPET) qualifiedName (WIDGET IDENTIFIER)? // DESCRIBE STYLING ON PAGE Module.Page [WIDGET name] | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index efc354c8d8..4736682301 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -285,11 +285,17 @@ anchorPoint : LPAREN NUMBER_LITERAL COMMA NUMBER_LITERAL RPAREN ; +// ADD VALUE IF NOT EXISTS / DROP VALUE IF EXISTS are the same idempotency +// guards as on ATTRIBUTE / INDEX above, and for the same reason: without them a +// script that adds an enumeration value is not re-runnable. The unguarded ADD +// errors on the second run and `exec` STOPS THERE, leaving every later statement +// unapplied — so one already-present value silently truncates the script. +// (ako/mxcli-rest FINDINGS #60) alterEnumerationAction - : ADD VALUE IDENTIFIER (CAPTION STRING_LITERAL)? + : ADD VALUE ifNotExists? IDENTIFIER (CAPTION STRING_LITERAL)? | RENAME VALUE IDENTIFIER TO IDENTIFIER | MODIFY VALUE IDENTIFIER CAPTION STRING_LITERAL - | DROP VALUE IDENTIFIER + | DROP VALUE ifExists? IDENTIFIER | SET COMMENT STRING_LITERAL ; diff --git a/mdl/grammar/domains/MDLPage.g4 b/mdl/grammar/domains/MDLPage.g4 index 23d75f7e9b..c003801d1c 100644 --- a/mdl/grammar/domains/MDLPage.g4 +++ b/mdl/grammar/domains/MDLPage.g4 @@ -249,7 +249,18 @@ snippetHeaderPropertyV3 // `placeholder { … }` block binds its widgets to that named layout // placeholder (issue #532 — pages over a layout with >1 placeholder). pageBodyV3 - : (widgetV3 | useFragmentRef | useBuildingBlockRef | placeholderBlockV3 | slotMarkerV3)* + // ORDER IS LOAD-BEARING since slices 2-3. widgetV3's last alternative is a + // generic (IDENTIFIER | keyword) widget type, and SLOT, PLACEHOLDER and USE + // are all in `keyword` — so with widgetV3 first, `slot content` parsed as a + // widget of type `slot` named `content`, and `placeholder Main { … }` as a + // widget named Main. Both still PARSED and still exited 0, which is why a + // diff of `mxcli check` output across all 515 example scripts did not show + // it; the damage is to the AST, not to the diagnostics. Two visitor unit + // tests caught it. + // + // The specific alternatives therefore go first, the same ordering fix + // widgetV3 already applies internally for `template for`. + : (useFragmentRef | useBuildingBlockRef | placeholderBlockV3 | slotMarkerV3 | widgetV3)* ; // SLOT [name] — a content placeholder inside a `define fragment` body. When the @@ -412,11 +423,38 @@ widgetTypeV3 // the dojo-based native Forms$DataGrid even on Mendix 11+; useful for // migrated projects that still have native datagrids on the page. | LEGACYDATAGRID + // Any widget with a definition, named by its MDL name — `htmlelement frame + // (...)`, `fileuploader up (...)`. Slice 2 of + // PROPOSAL_def_driven_widget_bodies.md (mendixlabs/mxcli#1036). + // + // The list above was never a capability boundary: cmd_pages_builder_v3.go's + // default branch already resolves widgetRegistry.Get(ToUpper(w.Type)) FIRST, + // and every .def.json declares an mdlName. Only ANTLR needed a token, so a + // widget mxcli could build was one MDL could not spell. + // + // ORDERED LAST so every enumerated type keeps winning its own alternative, + // and the widget's NAME is still a direct IDENTIFIER child of widgetV3 — + // this one is nested inside widgetTypeV3, so wCtx.IDENTIFIER() is unaffected. + // + // An unknown name is no longer a parse error; it is MDL-WIDGET25, which + // slice 0 added for exactly this reason. + | IDENTIFIER + // Slice 3: the same, for a container whose name lexes as a KEYWORD token. + // This is not defensive — it is the case that motivated the whole issue. + // `attribute` lexes as ATTRIBUTE and never as IDENTIFIER, so the + // alternative above cannot match `attribute a1 (...)`, which is the HTML + // Element object list the reporter could not write. + | keyword ; // V3 Widget properties: (Prop: Value, Prop: Value) +// The list may be EMPTY. `container c ()` is what an LLM writes when a widget +// needs no properties, and rejecting it gave a parse error at the `)` that read +// as though the widget itself were wrong. Bare `container c` already parsed, so +// this only removes an arbitrary difference between two spellings of the same +// thing (mendixlabs/mxcli#1036). widgetPropertiesV3 - : LPAREN widgetPropertyV3 (COMMA widgetPropertyV3)* RPAREN + : LPAREN (widgetPropertyV3 (COMMA widgetPropertyV3)*)? RPAREN ; widgetPropertyV3 diff --git a/mdl/grammar/domains/MDLSecurity.g4 b/mdl/grammar/domains/MDLSecurity.g4 index 7b7ab5eaa9..4addb9c221 100644 --- a/mdl/grammar/domains/MDLSecurity.g4 +++ b/mdl/grammar/domains/MDLSecurity.g4 @@ -115,8 +115,14 @@ dropDemoUserStatement : DROP DEMO USER STRING_LITERAL ; +// IN is optional before the module name, not just before the whole clause. +// `update security RestLab` used to reach the parser's error recovery, which +// consumed the name silently: the statement parsed as ONE statement with no +// error, `mxcli check` reported "Syntax OK", and the run went project-wide. +// A scope the author asked for and did not get is worse than a parse error. +// (mendixlabs/mxcli#1047) updateSecurityStatement - : UPDATE SECURITY (IN qualifiedName)? + : UPDATE SECURITY (IN? qualifiedName)? ; moduleRoleList diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index c4479d6975..b042833adf 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -628,7 +628,7 @@ keyword | CAPTION | CAPTIONPARAMS | CLASS | COLUMN | COLUMNS | CONTENT | CONTENTPARAMS | DATASOURCE | DEFAULT | DESIGNPROPERTIES | DESKTOPWIDTH | DISPLAY | DOCUMENTATION | EDITABLE | FILTER | FILTERTYPE | HEADER | FOOTER - | ICON | DARK | LABEL | ONCLICK | ONCHANGE | PARAMS | PASSING + | ICON | GLYPH | DARK | LABEL | ONCLICK | ONCHANGE | PARAMS | PASSING | PHONEWIDTH | TABLETWIDTH | READONLY | RENDERMODE | REQUIRED | NULLABLE | SELECTION | STYLE | STYLING | TABINDEX | TITLE | TOOLTIP | URL | POSITION | VISIBLE | WIDTH | HEIGHT | WIDGETTYPE diff --git a/mdl/linter/context.go b/mdl/linter/context.go index d0b08b4383..a2012d52ee 100644 --- a/mdl/linter/context.go +++ b/mdl/linter/context.go @@ -6,6 +6,7 @@ import ( "database/sql" "fmt" "iter" + "strings" "sync" "github.com/mendixlabs/mxcli/mdl/catalog" @@ -489,7 +490,7 @@ type Microflow struct { QualifiedName string ModuleName string Folder string - MicroflowType string // "Microflow", "Nanoflow" + MicroflowType string // "MICROFLOW", "NANOFLOW", "RULE" (as stored by the catalog) Description string ReturnType string ParameterCount int @@ -497,6 +498,33 @@ type Microflow struct { Complexity int // McCabe cyclomatic complexity } +// DocumentNoun is what to call this document in a lint message and in +// Location.DocumentType. +// +// Microflows() yields all three flow flavours, because they share one catalog +// table — so a rule that hardcodes "microflow" reports a nanoflow or a rule +// under the wrong doctype. That is how MPR002 came to say "Microflow 'Rule1' +// has no activities" about a rule, and the same about a nanoflow. +// +// An unrecognised type falls back to "microflow": a lint finding is worth more +// with an imprecise noun than not at all. +func (m Microflow) DocumentNoun() string { + switch m.MicroflowType { + case "NANOFLOW": + return "nanoflow" + case "RULE": + return "rule" + default: + return "microflow" + } +} + +// DocumentNounTitle is DocumentNoun capitalised, for a message that opens with it. +func (m Microflow) DocumentNounTitle() string { + n := m.DocumentNoun() + return strings.ToUpper(n[:1]) + n[1:] +} + // Microflows returns an iterator over all microflows (excluding system modules). func (ctx *LintContext) Microflows() iter.Seq[Microflow] { return func(yield func(Microflow) bool) { diff --git a/mdl/linter/rules/conv_error_handling.go b/mdl/linter/rules/conv_error_handling.go index a402d10372..b7a522c3de 100644 --- a/mdl/linter/rules/conv_error_handling.go +++ b/mdl/linter/rules/conv_error_handling.go @@ -81,7 +81,7 @@ func findUnhandledCalls(objects []microflows.MicroflowObject, mf linter.Microflo actionName, mf.ModuleName, mf.Name, act.ErrorHandlingType), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, @@ -156,7 +156,7 @@ func findContinueErrorHandling(objects []microflows.MicroflowObject, mf linter.M caption, mf.ModuleName, mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, @@ -176,7 +176,7 @@ func findContinueErrorHandling(objects []microflows.MicroflowObject, mf linter.M caption, mf.ModuleName, mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/conv_loop_commit.go b/mdl/linter/rules/conv_loop_commit.go index af15fa801c..a4796068de 100644 --- a/mdl/linter/rules/conv_loop_commit.go +++ b/mdl/linter/rules/conv_loop_commit.go @@ -59,12 +59,12 @@ func findCommitsInLoops(objects []microflows.MicroflowObject, mf linter.Microflo *violations = append(*violations, linter.Violation{ RuleID: r.ID(), Severity: r.DefaultSeverity(), - Message: fmt.Sprintf("Microflow '%s.%s' has a Commit action inside a loop. "+ + Message: fmt.Sprintf("%s '%s.%s' has a Commit action inside a loop. "+ "This causes N+1 database operations.", - mf.ModuleName, mf.Name), + mf.DocumentNounTitle(), mf.ModuleName, mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/conv_split_caption.go b/mdl/linter/rules/conv_split_caption.go index 04fb2f352d..0dc8837c1f 100644 --- a/mdl/linter/rules/conv_split_caption.go +++ b/mdl/linter/rules/conv_split_caption.go @@ -65,7 +65,7 @@ func findEmptySplitCaptions(objects []microflows.MicroflowObject, mf linter.Micr mf.ModuleName, mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/empty.go b/mdl/linter/rules/empty.go index 6255d6b9e0..2f3890c194 100644 --- a/mdl/linter/rules/empty.go +++ b/mdl/linter/rules/empty.go @@ -8,7 +8,12 @@ import ( "github.com/mendixlabs/mxcli/mdl/linter" ) -// EmptyMicroflowRule checks for microflows with no activities. +// EmptyMicroflowRule checks for flows with no activities. +// +// LintContext.Microflows() yields microflows, nanoflows and rules alike — they +// share one catalog table — so this reports on all three and names each by what +// it actually is. Hardcoding "microflow" is what made it announce +// "Microflow 'Rule1' has no activities" about a rule. type EmptyMicroflowRule struct{} // NewEmptyMicroflowRule creates a new empty microflow rule. @@ -22,7 +27,7 @@ func (r *EmptyMicroflowRule) Category() string { return "quality func (r *EmptyMicroflowRule) DefaultSeverity() linter.Severity { return linter.SeverityWarning } func (r *EmptyMicroflowRule) Description() string { - return "Checks for microflows that have no activities" + return "Checks for microflows, nanoflows and rules that have no activities" } // Check runs the empty microflow check. @@ -34,14 +39,14 @@ func (r *EmptyMicroflowRule) Check(ctx *linter.LintContext) []linter.Violation { violations = append(violations, linter.Violation{ RuleID: r.ID(), Severity: r.DefaultSeverity(), - Message: fmt.Sprintf("Microflow '%s' has no activities", mf.Name), + Message: fmt.Sprintf("%s '%s' has no activities", mf.DocumentNounTitle(), mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, - Suggestion: "Add activities or remove unused microflow", + Suggestion: fmt.Sprintf("Add activities or remove the unused %s", mf.DocumentNoun()), }) } } diff --git a/mdl/linter/rules/empty_test.go b/mdl/linter/rules/empty_test.go index 7a2660e069..777e55bc2d 100644 --- a/mdl/linter/rules/empty_test.go +++ b/mdl/linter/rules/empty_test.go @@ -101,3 +101,73 @@ func TestEmptyMicroflowRule_Metadata(t *testing.T) { t.Errorf("Category = %q, want quality", r.Category()) } } + +// TestEmptyMicroflowRule_NamesTheDocumentType is the test this rule did not +// have: LintContext.Microflows() yields microflows, nanoflows and rules alike +// (one catalog table, three doctypes), and MPR002 called every one of them a +// microflow — "Microflow 'Rule1' has no activities" about a rule. +// +// The fixtures use the catalog's own spellings, uppercase, rather than the +// title-case ones the older tests in this file use. That difference is the +// point: the old spellings never had to be right, because nothing read the +// column. Now that the noun is derived from it, a fixture that does not match +// what the catalog writes would test the fallback instead of the mapping. +func TestEmptyMicroflowRule_NamesTheDocumentType(t *testing.T) { + db := setupMicroflowsDB(t, [][]any{ + {"id1", "ACT_Process", "MyModule.ACT_Process", "MyModule", "", "MICROFLOW", "", "Void", 0, 0, 0}, + {"id2", "NF_Refresh", "MyModule.NF_Refresh", "MyModule", "", "NANOFLOW", "", "Void", 0, 0, 0}, + {"id3", "Rule1", "MyModule.Rule1", "MyModule", "", "RULE", "", "Boolean", 1, 0, 0}, + // An unknown type must still be reported, under the generic noun: a + // finding with an imprecise label beats no finding at all. + {"id4", "Mystery", "MyModule.Mystery", "MyModule", "", "SOMETHING_NEW", "", "Void", 0, 0, 0}, + }) + defer db.Close() + + violations := NewEmptyMicroflowRule().Check(linter.NewLintContextFromDB(db)) + if len(violations) != 4 { + t.Fatalf("expected 4 violations, got %d", len(violations)) + } + + byName := map[string]linter.Violation{} + for _, v := range violations { + byName[v.Location.DocumentName] = v + } + + for _, tc := range []struct{ doc, wantType, wantMessage string }{ + {"ACT_Process", "microflow", "Microflow 'ACT_Process' has no activities"}, + {"NF_Refresh", "nanoflow", "Nanoflow 'NF_Refresh' has no activities"}, + {"Rule1", "rule", "Rule 'Rule1' has no activities"}, + {"Mystery", "microflow", "Microflow 'Mystery' has no activities"}, + } { + v, ok := byName[tc.doc] + if !ok { + t.Errorf("no violation for %s", tc.doc) + continue + } + if v.Message != tc.wantMessage { + t.Errorf("%s: message = %q, want %q", tc.doc, v.Message, tc.wantMessage) + } + // The doctype also reaches the JSON and SARIF output as documentType, + // where a wrong value is not merely cosmetic. + if v.Location.DocumentType != tc.wantType { + t.Errorf("%s: documentType = %q, want %q", tc.doc, v.Location.DocumentType, tc.wantType) + } + } +} + +// TestDocumentNounCoversEveryCatalogType pins the mapping against the values +// the catalog actually inserts (mdl/catalog/builder_microflows.go), so adding a +// fourth flow flavour there fails here rather than silently reporting it as a +// microflow. +func TestDocumentNounCoversEveryCatalogType(t *testing.T) { + for stored, want := range map[string]string{ + "MICROFLOW": "microflow", + "NANOFLOW": "nanoflow", + "RULE": "rule", + } { + mf := linter.Microflow{MicroflowType: stored} + if got := mf.DocumentNoun(); got != want { + t.Errorf("DocumentNoun(%q) = %q, want %q", stored, got, want) + } + } +} diff --git a/mdl/linter/rules/flow_irreducible_graph.go b/mdl/linter/rules/flow_irreducible_graph.go index 14d65ff37b..413fcaaf2c 100644 --- a/mdl/linter/rules/flow_irreducible_graph.go +++ b/mdl/linter/rules/flow_irreducible_graph.go @@ -92,7 +92,7 @@ func (r *IrreducibleFlowGraphRule) violation(mf linter.Microflow, f microflowgra mf.ModuleName, mf.Name, what, f.Class, len(f.Overlap)), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/mpr008_overlapping_activities.go b/mdl/linter/rules/mpr008_overlapping_activities.go index ff9b5f3140..08e75a857f 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities.go +++ b/mdl/linter/rules/mpr008_overlapping_activities.go @@ -85,14 +85,14 @@ func (r *OverlappingActivitiesRule) Check(ctx *linter.LintContext) []linter.Viol RuleID: r.ID(), Severity: r.DefaultSeverity(), Message: fmt.Sprintf( - "Activities '%s' (%d,%d) and '%s' (%d,%d) overlap in microflow '%s.%s'. "+ + "Activities '%s' (%d,%d) and '%s' (%d,%d) overlap in %s '%s.%s'. "+ "Each MDL statement that creates a canvas activity needs its own @position annotation.", a.caption, a.x, a.y, b.caption, b.x, b.y, - mf.ModuleName, mf.Name, + mf.DocumentNoun(), mf.ModuleName, mf.Name, ), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/mpr011_loop_child_containment.go b/mdl/linter/rules/mpr011_loop_child_containment.go index 412c5179ec..0120275fc4 100644 --- a/mdl/linter/rules/mpr011_loop_child_containment.go +++ b/mdl/linter/rules/mpr011_loop_child_containment.go @@ -58,11 +58,12 @@ func (r *LoopChildContainmentRule) Check(ctx *linter.LintContext) []linter.Viola Severity: r.DefaultSeverity(), Message: fmt.Sprintf( "Activity '%s' at (%d,%d) lies outside the loop '%s' that contains it (box %dx%d) "+ - "in microflow '%s.%s'. The flow renders wrong in Studio Pro; mx check does not detect this.", - e.Child, e.ChildX, e.ChildY, e.Loop, e.BoxW, e.BoxH, mf.ModuleName, mf.Name), + "in %s '%s.%s'. The flow renders wrong in Studio Pro; mx check does not detect this.", + e.Child, e.ChildX, e.ChildY, e.Loop, e.BoxW, e.BoxH, + mf.DocumentNoun(), mf.ModuleName, mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/naming.go b/mdl/linter/rules/naming.go index 38a256e404..fcf758d048 100644 --- a/mdl/linter/rules/naming.go +++ b/mdl/linter/rules/naming.go @@ -93,7 +93,7 @@ func (r *NamingConventionRule) Check(ctx *linter.LintContext) []linter.Violation Message: fmt.Sprintf("Microflow name '%s' should use PascalCase with optional prefix (ACT_, SUB_, DS_, VAL_, SCH_, IVK_, BCO_, ACO_, BCR_, ACR_, BDE_, ADE_, BRO_, ARO_, OCH_, SE_, DL_, PWS_, ASU_, NAV_, LOGIN_)", mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/linter/rules/validation_feedback.go b/mdl/linter/rules/validation_feedback.go index ee95de3dbf..a71c7fe39d 100644 --- a/mdl/linter/rules/validation_feedback.go +++ b/mdl/linter/rules/validation_feedback.go @@ -70,7 +70,7 @@ func walkObjects(objects []microflows.MicroflowObject, mf linter.Microflow, r *V mf.ModuleName, mf.Name), Location: linter.Location{ Module: mf.ModuleName, - DocumentType: "microflow", + DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, diff --git a/mdl/types/navigation.go b/mdl/types/navigation.go index 4b55c2d277..10e71187fd 100644 --- a/mdl/types/navigation.go +++ b/mdl/types/navigation.go @@ -2,7 +2,11 @@ package types -import "github.com/mendixlabs/mxcli/model" +import ( + "strings" + + "github.com/mendixlabs/mxcli/model" +) // NavigationDocument represents a parsed navigation document. type NavigationDocument struct { @@ -54,11 +58,94 @@ type NavMenuItem struct { // Empty for no icon and for a glyph icon, which carries a numeric Code // instead. IconType keeps the storage $Type so a reader can tell the three // apart — DESCRIBE only round-trips Forms$IconCollectionIcon. - Icon string `json:"icon,omitempty"` - IconType string `json:"iconType,omitempty"` + Icon string `json:"icon,omitempty"` + IconType string `json:"iconType,omitempty"` + // IconCode is Forms$GlyphIcon's numeric Code — the ONLY thing that + // identifies a glyph icon, since it carries no qualified name. Without it a + // reader knows a glyph was there but not which one, so it can neither be + // re-emitted by DESCRIBE nor carried through a rewrite. + IconCode int `json:"iconCode,omitempty"` Items []*NavMenuItem `json:"items,omitempty"` } +// HasIcon reports whether the item carries an icon of ANY of the three kinds. +// +// Not `Icon != ""`: a glyph icon has a numeric code and no name, so the obvious +// test calls an item with a perfectly good icon iconless. MDL074 asks this +// question, and asking it the obvious way would have made the rule fire on every +// Studio Pro-authored menu. +func (m *NavMenuItem) HasIcon() bool { + if m == nil { + return false + } + switch MenuIconKindOf(m.IconType) { + case MenuIconNone: + return false + case MenuIconGlyph: + return true + default: + // A collection or image icon without a name is a malformed element, not + // an icon anyone can see. + return m.Icon != "" + } +} + +// MenuIconKind names which of Mendix's three icon elements a menu item carries. +// +// They are not variations on one shape: an icon-collection icon and an image +// icon each hold a qualified name (into an icon collection and an image +// collection respectively — different documents), while a glyph icon holds a +// numeric character code and no name at all. Treating them as one "icon string" +// is what made a rewrite silently convert a glyph into nothing. +type MenuIconKind string + +const ( + MenuIconNone MenuIconKind = "" + MenuIconCollection MenuIconKind = "collection" + MenuIconGlyph MenuIconKind = "glyph" + MenuIconImage MenuIconKind = "image" + // MenuIconUnknown is a stored $Type this build does not know. It is + // deliberately NOT MenuIconNone: reporting an unrecognised element as "no + // icon" is how a future fourth variant would get silently dropped by a + // rewrite, which is the bug this vocabulary exists to prevent. + MenuIconUnknown MenuIconKind = "unknown" +) + +// MenuIconKindOf maps a stored $Type onto the vocabulary. +// +// Matched on the suffix because the same element has two spellings: the +// metamodel calls it Pages$IconCollectionIcon and storage calls it +// Forms$IconCollectionIcon ("Form" was the original term for "Page"). A reader +// handing over either name must land on the same kind. +func MenuIconKindOf(iconType string) MenuIconKind { + switch { + case iconType == "": + return MenuIconNone + case strings.HasSuffix(iconType, "IconCollectionIcon"): + return MenuIconCollection + case strings.HasSuffix(iconType, "GlyphIcon"): + return MenuIconGlyph + case strings.HasSuffix(iconType, "ImageIcon"): + return MenuIconImage + } + return MenuIconUnknown +} + +// MenuIconStorageType is the inverse: the $Type a writer must emit for a kind. +// Empty for MenuIconNone (no Icon element at all) and for MenuIconUnknown, +// which a writer must never invent a name for. +func MenuIconStorageType(kind MenuIconKind) string { + switch kind { + case MenuIconCollection: + return "Forms$IconCollectionIcon" + case MenuIconGlyph: + return "Forms$GlyphIcon" + case MenuIconImage: + return "Forms$ImageIcon" + } + return "" +} + // MenuDocument is a standalone `Menus$MenuDocument` — a reusable menu that menu // widgets point at, stored as its own document rather than inside a navigation // profile. Atlas_Core ships two of them (Phone_Menu, Tablet_Menu). @@ -111,8 +198,15 @@ type NavMenuItemSpec struct { // as the same Forms$SignOutClientAction a button uses, so it needs no // target — which is why it is a flag rather than another name field. SignOut bool - // Icon is a qualified icon-collection name (Atlas_Core.Atlas.home). Empty - // means no icon, which serializes as a null Icon. - Icon string - Items []NavMenuItemSpec + // Icon is the qualified name for a collection or image icon + // (Atlas_Core.Atlas.home). Empty with IconKind None means no icon, which + // serializes as a null Icon. + Icon string + // IconKind selects which of the three icon elements to write. The spec used + // to carry a name and nothing else, so every icon became a + // Forms$IconCollectionIcon and a glyph turned into nothing on rewrite. + IconKind MenuIconKind + // IconCode is the glyph's numeric Code, meaningful only for MenuIconGlyph. + IconCode int + Items []NavMenuItemSpec } diff --git a/mdl/types/navigation_icon_test.go b/mdl/types/navigation_icon_test.go new file mode 100644 index 0000000000..2cc347554c --- /dev/null +++ b/mdl/types/navigation_icon_test.go @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: Apache-2.0 + +package types + +import "testing" + +// A menu item's icon is one of three Mendix elements, and mxcli could name only +// one of them. MenuIconKindOf maps the storage $Type onto a vocabulary so the +// readers, the writers, DESCRIBE and MDL074 all decide the same way instead of +// each doing its own strings.HasSuffix. +func TestMenuIconKindOf(t *testing.T) { + cases := []struct { + iconType string + want MenuIconKind + }{ + {"", MenuIconNone}, + {"Forms$IconCollectionIcon", MenuIconCollection}, + {"Forms$GlyphIcon", MenuIconGlyph}, + {"Forms$ImageIcon", MenuIconImage}, + // The metamodel spells these Pages$…; the storage name is Forms$… + // ("Form" was the original term for "Page"). Both must map, or a reader + // that hands over the metamodel name silently produces "no icon". + {"Pages$IconCollectionIcon", MenuIconCollection}, + {"Pages$GlyphIcon", MenuIconGlyph}, + {"Pages$ImageIcon", MenuIconImage}, + // Anything else is unknown rather than none: "none" would make a future + // fourth variant look like an absent icon and get silently dropped. + {"Forms$SomethingElse", MenuIconUnknown}, + } + for _, c := range cases { + if got := MenuIconKindOf(c.iconType); got != c.want { + t.Errorf("MenuIconKindOf(%q) = %q, want %q", c.iconType, got, c.want) + } + } +} + +// The inverse, used by the writers. Round-tripping the vocabulary back to the +// storage name is what lets one kind field drive both engines. +func TestMenuIconStorageType(t *testing.T) { + for _, k := range []MenuIconKind{MenuIconCollection, MenuIconGlyph, MenuIconImage} { + st := MenuIconStorageType(k) + if st == "" { + t.Errorf("no storage type for kind %q", k) + continue + } + if back := MenuIconKindOf(st); back != k { + t.Errorf("%q -> %q -> %q, want round trip", k, st, back) + } + } + if MenuIconStorageType(MenuIconNone) != "" { + t.Error("MenuIconNone must have no storage type — it is the absence of an Icon element") + } +} + +// HasIcon is what MDL074 asks. A glyph icon carries a numeric code and NO name, +// so a check written as `Icon == ""` reports an item that plainly has an icon. +func TestNavMenuItemHasIcon(t *testing.T) { + cases := []struct { + name string + item NavMenuItem + want bool + }{ + {"no icon", NavMenuItem{}, false}, + {"collection", NavMenuItem{IconType: "Forms$IconCollectionIcon", Icon: "Atlas_Core.Atlas.home"}, true}, + {"image", NavMenuItem{IconType: "Forms$ImageIcon", Icon: "MyMod.Images.logo"}, true}, + // The case that matters: a name-less icon that still IS an icon. + {"glyph", NavMenuItem{IconType: "Forms$GlyphIcon", IconCode: 57345}, true}, + } + for _, c := range cases { + if got := c.item.HasIcon(); got != c.want { + t.Errorf("%s: HasIcon() = %v, want %v", c.name, got, c.want) + } + } +} diff --git a/mdl/visitor/visitor_enumeration.go b/mdl/visitor/visitor_enumeration.go index 8624910675..19bace1aa1 100644 --- a/mdl/visitor/visitor_enumeration.go +++ b/mdl/visitor/visitor_enumeration.go @@ -57,16 +57,18 @@ func (b *Builder) ExitAlterEnumerationAction(ctx *parser.AlterEnumerationActionC caption = unquoteString(ctx.STRING_LITERAL().GetText()) } b.statements = append(b.statements, &ast.AlterEnumerationStmt{ - Name: name, - Operation: ast.AlterEnumAdd, - ValueName: ids[0].GetText(), - Caption: caption, + Name: name, + Operation: ast.AlterEnumAdd, + ValueName: ids[0].GetText(), + Caption: caption, + IfNotExists: ctx.IfNotExists() != nil, }) } else if ctx.DROP() != nil && ctx.VALUE() != nil && len(ids) >= 1 { b.statements = append(b.statements, &ast.AlterEnumerationStmt{ Name: name, Operation: ast.AlterEnumDrop, ValueName: ids[0].GetText(), + IfExists: ctx.IfExists() != nil, }) } else if ctx.RENAME() != nil && ctx.VALUE() != nil && len(ids) >= 2 { b.statements = append(b.statements, &ast.AlterEnumerationStmt{ diff --git a/mdl/visitor/visitor_navigation.go b/mdl/visitor/visitor_navigation.go index 460a529ed4..787761c1ca 100644 --- a/mdl/visitor/visitor_navigation.go +++ b/mdl/visitor/visitor_navigation.go @@ -3,8 +3,11 @@ package visitor import ( + "strconv" + "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/types" ) // ExitCreateNavigationStatement handles CREATE [OR REPLACE] NAVIGATION . @@ -93,29 +96,26 @@ func buildNavMenuItemDef(ctx parser.INavMenuItemDefContext) ast.NavMenuItemDef { item := ast.NavMenuItemDef{Caption: caption} - // Both the PAGE/MICROFLOW target and the ICON are qualifiedNames, so they - // arrive in one indexed list. The target always comes first when present; - // whatever remains after it is the icon. - names := c.AllQualifiedName() - next := 0 - switch { - case c.PAGE() != nil && len(names) > next: - built := buildQualifiedName(names[next]) - item.Page = &built - next++ - case c.MICROFLOW() != nil && len(names) > next: - built := buildQualifiedName(names[next]) - item.Microflow = &built - next++ + // The PAGE/MICROFLOW target is the item's only qualifiedName now that the + // icon is its own sub-rule — which is what removed the old positional + // bookkeeping, where the target and the icon shared one indexed list and the + // icon was "whatever remains". + if qn := c.QualifiedName(); qn != nil { + switch { + case c.PAGE() != nil: + built := buildQualifiedName(qn) + item.Page = &built + case c.MICROFLOW() != nil: + built := buildQualifiedName(qn) + item.Microflow = &built + } } - // SIGN_OUT names no target, so it consumes none of the qualifiedName list — - // which is why it is read separately rather than as a third switch arm. + // SIGN_OUT names no target, which is why it is read separately rather than + // as a third switch arm. if c.SIGN_OUT() != nil { item.SignOut = true } - if c.ICON() != nil && len(names) > next { - item.Icon = buildQualifiedName(names[next]).String() - } + applyNavMenuIcon(&item, c.NavMenuIcon()) // Recurse into sub-items (for MENU 'caption' (...)) for _, subCtx := range c.AllNavMenuItemDef() { @@ -125,3 +125,46 @@ func buildNavMenuItemDef(ctx parser.INavMenuItemDefContext) ast.NavMenuItemDef { return item } + +// applyNavMenuIcon reads the ICON clause onto the item. +// +// Mendix stores three different icon ELEMENTS, not three spellings of one +// value: a collection icon and an image icon each hold a qualified name (into an +// icon collection and an image collection — different documents), while a glyph +// icon holds a numeric character code and no name at all. The kind is recorded +// so the writer emits the right $Type; collapsing them onto one string is what +// made a rewrite turn a glyph into nothing. +// +// The bare form is the collection icon, which keeps every existing script +// meaning exactly what it did. +func applyNavMenuIcon(item *ast.NavMenuItemDef, ctx parser.INavMenuIconContext) { + if ctx == nil { + return + } + c, ok := ctx.(*parser.NavMenuIconContext) + if !ok { + return + } + switch { + case c.GLYPH() != nil: + item.IconKind = types.MenuIconGlyph + if n := c.NUMBER_LITERAL(); n != nil { + // A glyph code is a character code: whole, and small. A fractional or + // unparseable literal leaves the code at zero rather than guessing, + // and the writer refuses to emit a glyph without one. + if v, err := strconv.Atoi(n.GetText()); err == nil { + item.IconCode = v + } + } + case c.IMAGE() != nil: + item.IconKind = types.MenuIconImage + if qn := c.QualifiedName(); qn != nil { + item.Icon = buildQualifiedName(qn).String() + } + default: + item.IconKind = types.MenuIconCollection + if qn := c.QualifiedName(); qn != nil { + item.Icon = buildQualifiedName(qn).String() + } + } +} diff --git a/mdl/visitor/visitor_navigation_icon_test.go b/mdl/visitor/visitor_navigation_icon_test.go index f63430023b..2af4c1a05f 100644 --- a/mdl/visitor/visitor_navigation_icon_test.go +++ b/mdl/visitor/visitor_navigation_icon_test.go @@ -6,6 +6,7 @@ import ( "testing" "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/types" ) // navMenuItems parses a CREATE NAVIGATION with the given menu body and returns @@ -101,3 +102,88 @@ func TestNavMenuItem_IconWithoutATarget(t *testing.T) { t.Error("no target was written, so none must be built") } } + +// Mendix stores three different icon elements and MDL could name only one, so +// DESCRIBE emitted a comment for the other two and re-running its own output +// destroyed them. These are the two new forms. +// +// A parse test is not enough here: the slice 2-3 lesson is that a corpus diff of +// `check` output is blind to a construct that parses into the WRONG SHAPE. These +// assert the AST, which is what the writer reads. +func TestNavMenuItem_ParsesAGlyphIcon(t *testing.T) { + items := navMenuItems(t, `MENU ITEM 'Dashboard' PAGE M.Dash ICON GLYPH 57345;`) + if len(items) != 1 { + t.Fatalf("expected 1 item, got %d", len(items)) + } + if items[0].IconKind != types.MenuIconGlyph { + t.Errorf("IconKind = %q, want %q", items[0].IconKind, types.MenuIconGlyph) + } + if items[0].IconCode != 57345 { + t.Errorf("IconCode = %d, want 57345", items[0].IconCode) + } + if items[0].Icon != "" { + t.Errorf("Icon = %q, want empty — a glyph carries a code, not a name", items[0].Icon) + } + if items[0].Page == nil || items[0].Page.Name != "Dash" { + t.Error("the ICON clause displaced the PAGE target") + } +} + +func TestNavMenuItem_ParsesAnImageIcon(t *testing.T) { + items := navMenuItems(t, `MENU ITEM 'Dashboard' PAGE M.Dash ICON IMAGE MyMod.Images.logo;`) + if len(items) != 1 { + t.Fatalf("expected 1 item, got %d", len(items)) + } + if items[0].IconKind != types.MenuIconImage { + t.Errorf("IconKind = %q, want %q", items[0].IconKind, types.MenuIconImage) + } + if items[0].Icon != "MyMod.Images.logo" { + t.Errorf("Icon = %q", items[0].Icon) + } + if items[0].Page == nil || items[0].Page.Name != "Dash" { + t.Error("the ICON clause displaced the PAGE target") + } +} + +// The control, and the one that would break first: qualifiedName accepts a +// keyword as a name segment, so `ICON IMAGE …` also matches the BARE form with +// `image` read as the name. The bare form must keep meaning icon-collection, and +// the keyword-led form must not be swallowed by it. +func TestNavMenuItem_BareIconIsStillACollectionIcon(t *testing.T) { + items := navMenuItems(t, `MENU ITEM 'Home' PAGE M.Home ICON Atlas_Core.Atlas.home;`) + if items[0].IconKind != types.MenuIconCollection { + t.Errorf("IconKind = %q, want %q — the bare form is the collection icon", + items[0].IconKind, types.MenuIconCollection) + } + if items[0].Icon != "Atlas_Core.Atlas.home" { + t.Errorf("Icon = %q", items[0].Icon) + } + if items[0].IconCode != 0 { + t.Errorf("IconCode = %d, want 0", items[0].IconCode) + } +} + +// An item with no icon at all keeps the zero kind, which is what MDL074 reads. +func TestNavMenuItem_NoIconIsKindNone(t *testing.T) { + items := navMenuItems(t, `MENU ITEM 'Home' PAGE M.Home;`) + if items[0].IconKind != types.MenuIconNone { + t.Errorf("IconKind = %q, want none", items[0].IconKind) + } +} + +// A submenu takes the new forms too — it sits directly on the collapsed rail. +func TestNavMenuItem_SubMenuTakesAGlyphIcon(t *testing.T) { + items := navMenuItems(t, "MENU 'Admin' ICON GLYPH 100 (\n MENU ITEM 'Users' PAGE M.U ICON IMAGE M.I.u;\n);") + if len(items) != 1 { + t.Fatalf("expected 1 item, got %d", len(items)) + } + if items[0].IconKind != types.MenuIconGlyph || items[0].IconCode != 100 { + t.Errorf("submenu icon = (%q, %d), want (glyph, 100)", items[0].IconKind, items[0].IconCode) + } + if len(items[0].Items) != 1 { + t.Fatalf("sub-items = %d, want 1 — the icon clause ate the block", len(items[0].Items)) + } + if items[0].Items[0].IconKind != types.MenuIconImage { + t.Errorf("sub-item kind = %q, want image", items[0].Items[0].IconKind) + } +} diff --git a/mdl/visitor/visitor_page_generic_widget_test.go b/mdl/visitor/visitor_page_generic_widget_test.go new file mode 100644 index 0000000000..e5019566c3 --- /dev/null +++ b/mdl/visitor/visitor_page_generic_widget_test.go @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func buildOnePage(t *testing.T, src string) *ast.CreatePageStmtV3 { + t.Helper() + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors: %v\nsource:\n%s", errs, src) + } + if len(prog.Statements) != 1 { + t.Fatalf("got %d statements, want 1", len(prog.Statements)) + } + pg, ok := prog.Statements[0].(*ast.CreatePageStmtV3) + if !ok { + t.Fatalf("got %T, want *ast.CreatePageStmtV3", prog.Statements[0]) + } + return pg +} + +// Slice 2: a widget named by its own MDL name reaches the AST as that type, and +// is marked generic so the validator knows it must resolve to a definition. +func TestGenericWidgetTypeReachesTheAST(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + htmlelement frame (tagName: 'div') +}`) + if len(pg.Widgets) != 1 { + t.Fatalf("got %d widgets, want 1", len(pg.Widgets)) + } + w := pg.Widgets[0] + if w.Type != "htmlelement" { + t.Errorf("Type = %q, want %q", w.Type, "htmlelement") + } + if w.Name != "frame" { + t.Errorf("Name = %q, want \"frame\" — the name must still be the widget's own IDENTIFIER, "+ + "not swallowed by the generic type alternative", w.Name) + } + if !w.TypeIsGeneric { + t.Error("TypeIsGeneric = false; a name that is not an enumerated widget token must be " + + "marked generic, or MDL-WIDGET25 cannot tell a typo from a built-in") + } +} + +// The control: an enumerated widget keyword must NOT be marked generic, or every +// built-in would be required to resolve to a widget definition. +func TestEnumeratedWidgetTypeIsNotGeneric(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + container c1 (Class: 'x') +}`) + w := pg.Widgets[0] + if w.Type != "container" { + t.Fatalf("Type = %q, want container", w.Type) + } + if w.TypeIsGeneric { + t.Error("TypeIsGeneric = true for `container`, an enumerated widget token — " + + "the flag must come from which grammar alternative matched, not from a name lookup") + } +} + +// Slice 3: a container whose keyword lexes as a KEYWORD token, not IDENTIFIER. +// This is the construct from mendixlabs/mxcli#1036 — `attribute` inside an HTML +// Element — which failed with "mismatched input" before. +func TestKeywordContainerParsesInsideAWidgetBody(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + htmlelement frame (tagName: 'div') { + attribute a1 (attributeName: 'title') + event e1 (eventName: 'onClick') + } +}`) + w := pg.Widgets[0] + if len(w.Children) != 2 { + t.Fatalf("got %d children, want 2 (attribute, event)", len(w.Children)) + } + for i, want := range []string{"attribute", "event"} { + if w.Children[i].Type != want { + t.Errorf("child %d Type = %q, want %q", i, w.Children[i].Type, want) + } + if !w.Children[i].TypeIsGeneric { + t.Errorf("child %d (%s) not marked generic", i, want) + } + } +} + +// pageBodyV3's alternative ORDER is load-bearing since slices 2-3, and this is +// the trap that a diff of `mxcli check` output cannot detect. +// +// SLOT, PLACEHOLDER and USE are all inside the `keyword` rule, so widgetV3's +// generic alternative can match them. With widgetV3 ordered first, `slot body` +// parsed as a widget of type `slot` named `body`, and `placeholder Main { … }` +// as a widget named Main. Both still parsed and `check` still exited 0 — the +// damage is to the AST, not to the diagnostics, which is why 515 example +// scripts showed zero difference while two visitor tests failed. +// +// Anyone reordering pageBodyV3 for tidiness reintroduces it silently. +func TestSpecificPageBodyFormsWinOverTheGenericWidget(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + placeholder Main { + dynamictext t (Content: 'hi') + } +}`) + if len(pg.Placeholders) != 1 { + t.Fatalf("placeholder blocks = %d, want 1 — `placeholder` was swallowed by the generic "+ + "widget alternative; the specific alternatives must precede widgetV3 in pageBodyV3", + len(pg.Placeholders)) + } + if len(pg.Widgets) != 0 { + t.Errorf("bare widgets = %d, want 0 — the placeholder block was parsed as a widget", len(pg.Widgets)) + } +} + +func TestSlotMarkerWinsOverTheGenericWidget(t *testing.T) { + prog, errs := Build(`DEFINE FRAGMENT Card AS { + CONTAINER wrap (Class: 'c') { + SLOT content + } + };`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 1 { + t.Fatalf("got %d statements, want 1", len(prog.Statements)) + } + // `slot` is in the `keyword` rule, so the generic widget alternative can + // match it. The fragment must still see a slot, not a widget named `body`. + if got := fmt.Sprintf("%#v", prog.Statements[0]); strings.Contains(strings.ToLower(got), `type:"slot"`) { + t.Errorf("`slot body` became a widget of type slot — slotMarkerV3 must precede widgetV3 in pageBodyV3:\n%s", got) + } +} diff --git a/mdl/visitor/visitor_page_v3.go b/mdl/visitor/visitor_page_v3.go index 4713b5a2d6..099dfef20c 100644 --- a/mdl/visitor/visitor_page_v3.go +++ b/mdl/visitor/visitor_page_v3.go @@ -552,6 +552,18 @@ func buildWidgetV3(ctx parser.IWidgetV3Context, b *Builder) *ast.WidgetV3 { widget.Properties["WidgetType"] = unquoteString(wCtx.STRING_LITERAL().GetText()) } else if typeCtx := wCtx.WidgetTypeV3(); typeCtx != nil { widget.Type = strings.ToLower(typeCtx.GetText()) + // Which alternative matched, taken from the parse tree rather than by + // comparing the text against a list of known widget names. A generic + // type must resolve to a widget definition; an enumerated one is a + // built-in. See ast.WidgetV3.TypeIsGeneric. + // Both generic alternatives count. IDENTIFIER covers `htmlelement` + // (slice 2); Keyword covers a container whose name lexes as a keyword + // token, such as `attribute` (slice 3) — the case that motivated the + // issue. An enumerated widget type is a direct token alternative of + // widgetTypeV3 and matches neither accessor. + if typeCtx.IDENTIFIER() != nil || typeCtx.Keyword() != nil { + widget.TypeIsGeneric = true + } } // Get required identifier. The name may be quoted (QUOTED_IDENTIFIER) when it diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 88e9b6a460..8d042a31de 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -1011,6 +1011,24 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { return } + // DESCRIBE WIDGET — a widget DEFINITION, not a + // document, so there is no qualified name. The STYLING branch above + // returns before this, but WIDGET also appears there (DESCRIBE STYLING … + // WIDGET name), so the guard stays explicit rather than resting on order. + if ctx.WIDGET() != nil && ctx.STYLING() == nil { + name := "" + if lit := ctx.STRING_LITERAL(); lit != nil { + name = unquoteString(lit.GetText()) + } else if id := ctx.IdentifierOrKeyword(0); id != nil { + name = id.GetText() + } + b.statements = append(b.statements, &ast.DescribeStmt{ + ObjectType: ast.DescribeWidget, + Name: ast.QualifiedName{Name: name}, + }) + return + } + // Handle DESCRIBE NAVIGATION [profile] if ctx.NAVIGATION() != nil { stmt := &ast.DescribeStmt{ObjectType: ast.DescribeNavigation} diff --git a/sdk/mpr/parser_misc.go b/sdk/mpr/parser_misc.go index 6eeb70b12a..bd9adcb702 100644 --- a/sdk/mpr/parser_misc.go +++ b/sdk/mpr/parser_misc.go @@ -578,6 +578,10 @@ func parseNavMenuItem(raw map[string]any) *NavMenuItem { if icon, ok := raw["Icon"].(map[string]any); ok { mi.IconType = extractString(icon["$Type"]) mi.Icon = extractString(icon["Image"]) + // The glyph's Code identifies WHICH glyph; without it a caller knows one + // was there and nothing more, so it cannot be re-emitted or carried + // through a rewrite. + mi.IconCode = extractInt(icon["Code"]) } // Extract action type and target from Action diff --git a/sdk/mpr/writer_navigation.go b/sdk/mpr/writer_navigation.go index 57c3b3e305..44a90559f1 100644 --- a/sdk/mpr/writer_navigation.go +++ b/sdk/mpr/writer_navigation.go @@ -304,7 +304,7 @@ func buildMenuItemBson(mi NavMenuItemSpec) bson.D { {Key: "Action", Value: buildMenuAction(mi)}, {Key: "AlternativeText", Value: nil}, {Key: "Caption", Value: buildCaptionBson(mi.Caption)}, - {Key: "Icon", Value: buildMenuIconBson(mi.Icon)}, + {Key: "Icon", Value: buildMenuIconBson(mi)}, } // Sub-items @@ -327,21 +327,46 @@ func buildMenuItemBson(mi NavMenuItemSpec) bson.D { // the widget icon path already proven in issue #602. // // Two sibling variants exist in the same document — Forms$GlyphIcon{Code: int} -// and Forms$ImageIcon{Image: QN} — and are deliberately NOT emitted here. Both -// carry a different payload shape, and ImageIcon's qualified name is -// indistinguishable from an IconCollectionIcon's without resolving which -// collection document it lands in. Guessing between polymorphic variants is the -// failure mode that produces a document mxbuild accepts and Studio Pro cannot -// open. -func buildMenuIconBson(icon string) interface{} { - if icon == "" { +// and Forms$ImageIcon{Image: QN}. They used to be excluded because a name alone +// cannot tell an image icon from a collection icon without resolving which +// document it lands in, and guessing between polymorphic variants is the failure +// mode that produces a document mxbuild accepts and Studio Pro cannot open. +// +// Nothing is guessed now: the KIND is carried explicitly, from the author's own +// `icon image …` / `icon glyph …` or from the kind the reader saw in storage. So +// all three are emitted, and the branch is a dispatch rather than an inference. +// +// Excluding them was not neutral. `create or replace navigation` is a full +// replacement, so an icon the writer would not emit was an icon the statement +// DELETED — measured on testdata/expr-checker, exec of DESCRIBE's own output +// destroyed a glyph icon at exit 0. +func buildMenuIconBson(spec NavMenuItemSpec) interface{} { + kind := spec.IconKind + if kind == types.MenuIconNone && spec.Icon != "" { + // A spec built before the kind existed carries a name and nothing else, + // and that name has only ever meant an icon-collection icon. + kind = types.MenuIconCollection + } + storage := types.MenuIconStorageType(kind) + if storage == "" { return nil } - return bson.D{ + doc := bson.D{ {Key: "$ID", Value: idToBsonBinary(generateUUID())}, - {Key: "$Type", Value: "Forms$IconCollectionIcon"}, - {Key: "Image", Value: icon}, + {Key: "$Type", Value: storage}, + } + if kind == types.MenuIconGlyph { + // A glyph with no code identifies no glyph. Emit no icon rather than an + // element nobody can see. + if spec.IconCode == 0 { + return nil + } + return append(doc, bson.E{Key: "Code", Value: int32(spec.IconCode)}) + } + if spec.Icon == "" { + return nil } + return append(doc, bson.E{Key: "Image", Value: spec.Icon}) } // buildCaptionBson builds a Texts$Text BSON document with a single en_US translation. diff --git a/sdk/mpr/writer_navigation_icon_test.go b/sdk/mpr/writer_navigation_icon_test.go index 90efc9f593..970ec70b97 100644 --- a/sdk/mpr/writer_navigation_icon_test.go +++ b/sdk/mpr/writer_navigation_icon_test.go @@ -3,6 +3,7 @@ package mpr import ( + "github.com/mendixlabs/mxcli/mdl/types" "testing" "go.mongodb.org/mongo-driver/bson" @@ -30,7 +31,7 @@ func navIconEntry(d bson.D, key string) (interface{}, bool) { // Studio Pro-authored reference: every menu icon in ako/mxcli-ledger's // navigation document is Forms$IconCollectionIcon{Image: "Atlas_Core.Atlas.…"}. func TestBuildMenuIconBson_UsesTheFormsStorageName(t *testing.T) { - got := buildMenuIconBson("Atlas_Core.Atlas.align-center") + got := buildMenuIconBson(NavMenuItemSpec{Icon: "Atlas_Core.Atlas.align-center"}) d, ok := got.(bson.D) if !ok { t.Fatalf("expected a bson.D, got %T", got) @@ -59,8 +60,8 @@ func TestBuildMenuIconBson_UsesTheFormsStorageName(t *testing.T) { // No icon must stay a null, not an empty element: an IconCollectionIcon with a // blank Image is a dangling reference, where absent is the modelled default. func TestBuildMenuIconBson_EmptyNameStaysNull(t *testing.T) { - if got := buildMenuIconBson(""); got != nil { - t.Errorf("buildMenuIconBson(\"\") = %v, want nil", got) + if got := buildMenuIconBson(NavMenuItemSpec{}); got != nil { + t.Errorf("buildMenuIconBson(empty spec) = %v, want nil", got) } } @@ -203,3 +204,69 @@ func TestParseNavMenuItem_NoIconReadsAsNone(t *testing.T) { t.Errorf("Icon/IconType = (%q, %q), want both empty", mi.Icon, mi.IconType) } } + +// All three icon elements are written now. Only the collection variant used to +// be, and because `create or replace navigation` is a full replacement, an icon +// the writer would not emit was an icon the statement DELETED — measured on +// testdata/expr-checker, exec of DESCRIBE's own output destroyed a glyph icon at +// exit 0. +func TestBuildMenuIconBson_Glyph(t *testing.T) { + got, ok := buildMenuIconBson(NavMenuItemSpec{IconKind: types.MenuIconGlyph, IconCode: 57345}).(bson.D) + if !ok { + t.Fatal("a glyph icon produced no document") + } + m := bsonDToMap(got) + if m["$Type"] != "Forms$GlyphIcon" { + t.Errorf("$Type = %v, want Forms$GlyphIcon", m["$Type"]) + } + if m["Code"] != int32(57345) { + t.Errorf("Code = %#v, want int32(57345) — Mendix stores the character code as an int32", m["Code"]) + } + if _, hasImage := m["Image"]; hasImage { + t.Error("a glyph icon must not carry an Image; it has no qualified name") + } +} + +func TestBuildMenuIconBson_Image(t *testing.T) { + got, ok := buildMenuIconBson(NavMenuItemSpec{IconKind: types.MenuIconImage, Icon: "MyMod.Images.logo"}).(bson.D) + if !ok { + t.Fatal("an image icon produced no document") + } + m := bsonDToMap(got) + if m["$Type"] != "Forms$ImageIcon" { + t.Errorf("$Type = %v, want Forms$ImageIcon", m["$Type"]) + } + if m["Image"] != "MyMod.Images.logo" { + t.Errorf("Image = %v", m["Image"]) + } +} + +// The control that keeps the dispatch honest: a name with NO kind still means an +// icon-collection icon. Every script written before the kind existed carries +// exactly that, so treating it as "unknown" would silently drop every icon in +// the corpus. +func TestBuildMenuIconBson_BareNameIsStillACollectionIcon(t *testing.T) { + got, ok := buildMenuIconBson(NavMenuItemSpec{Icon: "Atlas_Core.Atlas.home"}).(bson.D) + if !ok { + t.Fatal("a bare name produced no document") + } + if bsonDToMap(got)["$Type"] != "Forms$IconCollectionIcon" { + t.Errorf("$Type = %v, want Forms$IconCollectionIcon", bsonDToMap(got)["$Type"]) + } +} + +// A glyph with no code identifies no glyph, and an element with no Code renders +// as a blank where an icon should be. Emit nothing instead. +func TestBuildMenuIconBson_GlyphWithoutACodeIsNoIcon(t *testing.T) { + if got := buildMenuIconBson(NavMenuItemSpec{IconKind: types.MenuIconGlyph}); got != nil { + t.Errorf("got %v, want nil", got) + } +} + +func bsonDToMap(d bson.D) map[string]any { + m := make(map[string]any, len(d)) + for _, e := range d { + m[e.Key] = e.Value + } + return m +} diff --git a/testdata/expr-checker/.gitignore b/testdata/expr-checker/.gitignore index 9ea33a1009..f2018de286 100644 --- a/testdata/expr-checker/.gitignore +++ b/testdata/expr-checker/.gitignore @@ -23,6 +23,9 @@ /mprcontents/mprjournal* .claude/settings.local.json +# Derived from widgets/*.mpk and rewritten by `refresh catalog full` +# (executor.RegenerateWidgetDocs), not authored here. +.claude/skills/widgets/ mxcli.exe mxcli .mxcli