diff --git a/AGENTS.md b/AGENTS.md index a4b5ceffa0c..0195d90495d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -110,6 +110,18 @@ tests/ ### Development Workflow +Read the area guide before changing a compiler subsystem: + +- [`compiler/syntax/README.md`](compiler/syntax/README.md) for parsing, + printing, and JSX transformation +- [`compiler/ml/README.md`](compiler/ml/README.md) for the type checker and + typed tree +- [`compiler/core/README.md`](compiler/core/README.md) for Lambda, Lam, and + JavaScript generation +- [`analysis/README.md`](analysis/README.md) for editor analysis +- [`rewatch/README.md`](rewatch/README.md) for the build system +- [`tools/README.md`](tools/README.md) for `rescript-tools` + 1. **Understand which layer you're working on:** - **Syntax layer** (`compiler/syntax/`): Parsing and surface syntax - **ML layer** (`compiler/ml/`): Type checking and AST transformations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5fc469fffec..2117cbeffb8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,6 +10,24 @@ We tried to keep the installation process as simple as possible. In case you are Happy hacking! +## Repository guides + +This document covers repository-wide setup and workflow. Start with the guide +for the compiler area you are changing: + +- [parser, printer, and JSX transformation](compiler/syntax/README.md) +- [type checker and typed tree](compiler/ml/README.md) +- [Lambda, Lam, and JavaScript generation](compiler/core/README.md) +- [editor analysis](analysis/README.md) +- [build system](rewatch/README.md) +- [`rescript-tools`](tools/README.md) + +Public language and library documentation belongs on the +[ReScript website](https://github.com/rescript-association/rescript-lang.org). +The guides in this repository document its current implementation and +contributor workflows. Detailed caller contracts belong in module interfaces; +algorithm and representation invariants belong beside their implementation. + ## Setup > Most of our contributors are working on Apple machines, so all our instructions are currently macOS / Linux centric. Contributions for Windows development welcome! diff --git a/analysis/README.md b/analysis/README.md index 813e4c7f6f5..fc8711eee0d 100644 --- a/analysis/README.md +++ b/analysis/README.md @@ -1,106 +1,82 @@ -# Analysis Library and Binary - -This subfolder builds a private command line binary used by the plugin to power a few functionalities such as jump to definition, hover and autocomplete. - -The binary reads the `.cmt` and `.cmti` files and analyses them. - -For installation & build instructions, see the main CONTRIBUTING.md. - -## Overview - -See main CONTRIBUTING.md's repo structure. Check out `test.sh` (invoked through `make test`) to see the snapshots testing workflow stored in `tests/`. - -## Usage - -```shell +# Editor analysis + +The analysis executable powers editor features such as completion, hover, +references, semantic tokens, and code actions. It reads compiler-produced +`.cmt` and `.cmti` files, so an analysis binary and the project artifacts it +inspects must be built with compatible compiler representations. + +## Code map + +- [`bin/main.ml`](bin/main.ml) starts the `rescript-editor-analysis` command. +- [`src/commands.ml`](src/commands.ml) dispatches commands, including the + source-annotated test command. +- `src/completion_*.ml` implements the completion frontend, context-specific + completion logic, and result conversion. +- [`src/hover.ml`](src/hover.ml), [`src/references.ml`](src/references.ml), + [`src/semantic_tokens.ml`](src/semantic_tokens.ml), and + [`src/code_actions.ml`](src/code_actions.ml) own the corresponding features. +- [`src/cmt.ml`](src/cmt.ml), [`src/process_cmt.ml`](src/process_cmt.ml), and + [`src/process_extra.ml`](src/process_extra.ml) are the main typed-artifact + boundary. Shared typed-tree definitions and traversal utilities live under + `compiler/ml`. +- [`reactive/README.md`](reactive/README.md) documents the reactive analysis + library. [`reanalyze/README.md`](reanalyze/README.md) covers Reanalyze, which + is a separate analysis pipeline in this directory. + +Run the binary from the repository root: + +```sh dune exec -- rescript-editor-analysis --help ``` -Add verbose logging via: - -```shell -dune exec -- rescript-editor-analysis debug-dump verbose test -``` - -## History - -This project is based on a fork of [Reason Language Server](https://github.com/jaredly/reason-language-server). - ## Tests -### Prerequisites +Build the compiler and runtime, then run the repository target: -- Ensure the compiler is built (`make build` in the repository root). -- Ensure the library is built (`make lib` in the repository root). +```sh +make lib +make test-analysis +``` -### Running the Tests +The target runs the suites under `tests/analysis_tests/`, including the main +snapshot suite and focused projects for generic JSX, incremental type checking, +namespaced references, and source-directory dependencies. -Run `make test` in `tests/analysis_tests/tests`. +The main suite uses directives embedded in ReScript comments. For example: -### Key Concept +```rescript +let value = 5 +// value. +// ^com +``` -The tests in the `tests/analysis_tests/tests` folder are based on the `dune exec -- rescript-editor-analysis test` command. This special subcommand processes a file and executes specific editor analysis functionality based on special syntax found in code comments. +`^com` asks the test command to compute completion at that position. See the +directive match in `analysis/src/commands.ml` for the current set. Tests compile +a temporary source file and compare command output with checked-in snapshots. +After an intentional change, inspect every updated snapshot rather than +accepting the directory wholesale. -Consider the following code: +To inspect one test while developing: -```res -let a = 5 -// a. -// ^com +```sh +dune exec -- rescript-editor-analysis test tests/analysis_tests/tests/src/CompletePrioritize1.res ``` -After building the ReScript project (**⚠️ this is a requirement**), you can execute `dune exec -- rescript-editor-analysis test Sample.res`, and completion will be performed for the cursor position indicated by `^`. The `com` directive requests completion. To see other commands, check out the pattern match in the `test` function in [Commands.ml](./src/Commands.ml). - -> [!WARNING] -> Ensure there are no spaces in the code comments, as the commands are captured by a regular expression that expects spaces and not tabs! +Use paths from the repository root, and ensure the test project has first been +built with the local compiler. -Here’s how it works: once a command is found in a comment, a copy of the source file is created inside a temporary directory, where the line above `^com` is uncommented. The corresponding analysis functionality is then processed, typically with `~debug:true`. With debug enabled, code paths like +## Changing typed compiler representations -```ml -if Debug.verbose () then - print_endline "[complete_typed_value]--> Tfunction #other"; -``` +Editor analysis consumes typedtree nodes and compiler type representations +directly. When adding or changing a parsetree or typedtree node: -will print to stdout. This is helpful for observing what happens during the analysis. - -When you run `make test` (from the `tests/analysis_tests` folder), `dune exec -- rescript-editor-analysis test ` will be executed for each `*.res` file in `analysis/tests/src`. The stdout will be compared to the corresponding `analysis/tests/src/expected` file. If `git diff` indicates changes, `make test` will fail, as these differences might be unintentional. - -## Testing on Your Own Projects - -To use a local version of `rescript-editor-analysis`, the targeted project needs to be compiled with the local compiler. - -Install your local ReScript with `npm i /path/to/your-local-rescript-repo`. -Reinstall the dependencies and run `npx rescript` in your project. This ensures the project is compiled with the same compiler version that the `rescript-editor-analysis` will process. - -## Debugging - -It is possible to debug `analysis` via [ocamlearlybird](https://github.com/hackwaly/ocamlearlybird). - -1. Install `opam install earlybird`. -2. Install the [earlybird extension](https://marketplace.visualstudio.com/items?itemName=hackwaly.ocamlearlybird). -3. Create a launch configuration (`.vscode/launch.json`): - -```json -{ - "version": "0.2.0", - "configurations": [ - { - "name": "Debug analysis", - "type": "ocaml.earlybird", - "request": "launch", - "program": "${workspaceFolder}/_build/default/analysis/bin/main.bc", - "stopOnEntry": true, - "cwd": "/projects/your-project", - "env": { - "CAML_LD_LIBRARY_PATH": "${workspaceFolder}/_build/default/compiler/ext" - }, - "arguments": [ - "test", - "src/Main.res" - ] - } - ] -} -``` +1. search this directory for matches on the surrounding constructors, rather + than relying only on exhaustiveness warnings; +2. check completion, hover/type printing, references, document symbols, + semantic tokens, code actions, and interface generation as applicable; +3. update both positive results and recovery behavior for incomplete source; +4. run `make test-analysis` in addition to compiler tests. -The `CAML_LD_LIBRARY_PATH` environment variable is required to tell OCaml where `dllext_stubs.so` can be loaded from. \ No newline at end of file +Keep compiler representation contracts in the owning compiler `.mli` files. +Put analysis-specific assumptions beside the analysis code that relies on them, +and use this guide only for navigation and cross-cutting workflow. diff --git a/analysis/reactive/README.md b/analysis/reactive/README.md index 9f55e57eff1..0eb0a47eee9 100644 --- a/analysis/reactive/README.md +++ b/analysis/reactive/README.md @@ -98,6 +98,11 @@ This prevents issues like: - Anti-joins seeing partial data (e.g., refs without matching decls) - Multi-level unions causing spurious additions/removals +The `fixpoint` implementation maintains exact reachability when roots and +edges are added or removed. See +[IncrementalFixpointReport.md](IncrementalFixpointReport.md) for the algorithm, +its invariants, and the limits of its evaluation data. + ## Usage in Reanalyze This library powers the reactive dead code analysis in reanalyze: @@ -106,4 +111,3 @@ This library powers the reactive dead code analysis in reanalyze: - `ReactiveMerge`: Merges per-file data into global collections - `ReactiveLiveness`: Computes live declarations via fixpoint - `ReactiveSolver`: Generates dead code issues reactively - diff --git a/analysis/reanalyze/ARCHITECTURE.md b/analysis/reanalyze/ARCHITECTURE.md index ae047658133..38e023a7621 100644 --- a/analysis/reanalyze/ARCHITECTURE.md +++ b/analysis/reanalyze/ARCHITECTURE.md @@ -126,16 +126,20 @@ AnalysisResult.get_issues analysis_result --- -## Incremental Updates (Future) +## Incremental updates in the non-reactive pipeline -The architecture enables incremental updates when a file changes: +The phase boundaries permit the non-reactive pipeline to update one file's +input without retaining mutable per-file analysis state: 1. Re-run Phase 1 for changed file only → new `file_data` 2. Replace in `file_data` map (keyed by filename) 3. Re-run Phase 2 (merge) - fast, pure function 4. Re-run Phase 3 (solve) - fast, pure function -The key insight: **immutable data structures enable safe incremental updates** - you can swap one file's data without affecting others. +Immutable phase outputs allow one file's data to be replaced without mutating +the retained outputs for other files. The current reactive pipeline below goes +further by propagating deltas through derived collections rather than rerunning +the complete merge and solve phases. --- @@ -334,4 +338,3 @@ Use `-timing` flag to see per-node statistics: | `Issue` | Issue type definitions | | `Log_` | Phase 4: Logging output | | `ReactiveSolver` | Reactive dead_decls → issues computation | - diff --git a/analysis/reanalyze/DEADCODE_REFACTOR_PLAN.md b/analysis/reanalyze/DEADCODE_REFACTOR_PLAN.md deleted file mode 100644 index 107493bb281..00000000000 --- a/analysis/reanalyze/DEADCODE_REFACTOR_PLAN.md +++ /dev/null @@ -1,679 +0,0 @@ -## Dead Code Analysis – Pure Pipeline Refactor Plan - -**Goal**: Turn the reanalyze dead code analysis into a transparent, effect-free pipeline where: -- Analysis is a pure function from inputs → results -- Global mutable state is eliminated -- Side effects (logging, file I/O) live at the edges -- Processing files in different orders gives the same results -- **Incremental analysis is possible** - can reprocess one file without redoing everything - -**Why?** The current architecture makes: -- Incremental/reactive analysis impossible (can't reprocess one file) -- Testing hard (global state persists between tests) -- Parallelization impossible (shared mutable state) -- Reasoning difficult (order-dependent hidden mutations) - ---- - -## Key Design Principles - -### 1. Local mutable state during AST processing, immutable after - -**AST processing phase** (per-file): -- Uses local mutable state for performance (hashtables, etc.) -- Returns **immutable** `file_data` when done -- This phase is inherently sequential per-file - -**Analysis phase** (project-wide): -- Works only with **immutable data structures** -- Must be parallelizable, reorderable -- Static guarantees from this point on - -```ocaml -(* AST processing: local mutable state OK, returns immutable *) -let process_file config cmt_infos : file_data = - let local_state = Hashtbl.create 256 in (* local mutable *) - ... traverse AST, mutate local_state ... - freeze_to_file_data local_state (* return immutable *) - -(* Analysis: immutable in, immutable out - parallelizable *) -let solve_deadness config (files : file_data list) : analysis_result = - ... pure computation on immutable data ... -``` - -### 2. Clear phase boundaries - -| Phase | Input | Mutability | Output | Parallelizable? | -|-------|-------|------------|--------|-----------------| -| **AST processing** | cmt file | Local mutable OK | Immutable `file_data` | Per-file yes | -| **Merge** | `file_data list` | None | Immutable merged view | Yes | -| **Analysis** | Merged view | None | Immutable `result` | Yes | -| **Reporting** | `result` | I/O side effects | None | N/A | - -### 3. Enable incremental updates - -When file F changes: -1. Re-run AST processing for F only → new `file_data` -2. Replace in `file_data` map (keyed by filename) -3. Re-run merge and analysis (on immutable data) - -The key is that **immutable data structures enable safe incremental updates** - -you can swap one file's data without affecting others. - ---- - -## Architecture - -See [ARCHITECTURE.md](./ARCHITECTURE.md) for the full architecture documentation with diagrams. - ---- - -## Current Problems (What We're Fixing) - -### P1: Global "current file" context -**Problem**: `Common.currentSrc`, `currentModule`, `currentModuleName` are global refs set before processing each file. Every function implicitly depends on "which file are we processing right now?". This makes it impossible to process multiple files concurrently or incrementally. - -**Used by**: `DeadCommon.addDeclaration_`, `DeadType.addTypeDependenciesAcrossFiles`, `DeadValue` path construction. - -**Status**: ✅ FIXED in Task 1 - explicit `file_context` now threaded through all analysis functions. - -### P2: Global analysis tables -**Problem**: All analysis results accumulate in global hashtables: -- `DeadCommon.decls` - all declarations -- `ValueReferences.table` - all value references -- `TypeReferences.table` - all type references -- `FileReferences.table` - cross-file dependencies - -**Impact**: Can't analyze a subset of files without reanalyzing everything. Can't clear state between test runs without module reloading. - -### P3: Cross-file processing queues -**Problem**: Several analyses use global queues that get "flushed" later: -- `DeadOptionalArgs.delayedItems` - cross-file optional arg analysis → DELETED (now `CrossFileItems`) -- `DeadException.delayedItems` - cross-file exception checks → DELETED (now `CrossFileItems`) -- `DeadType.TypeDependencies.delayedItems` - per-file type deps (already handled per-file) -- `ProcessDeadAnnotations.positionsAnnotated` - annotation tracking - -**Additional problem**: `positionsAnnotated` mixes **input** (source annotations from AST) with **output** (positions the solver determines are dead). The solver mutates this during analysis, violating purity. - -**Impact**: Order-dependent. Processing files in different orders can give different results because queue processing happens at arbitrary times. Mixing input/output prevents incremental analysis. - -### P4: Global configuration reads -**Problem**: Analysis code directly reads `!Common.Cli.debug`, `RunConfig.runConfig.transitive`, etc. scattered throughout. Can't run analysis with different configs without mutating globals. - -**Status**: ✅ FIXED in Task 2 - explicit `config` now threaded through all analysis functions. - -### P5: Side effects mixed with analysis -**Problem**: Analysis functions directly call: -- `Log_.warning` - logging -- `EmitJson` - JSON output -- ~~`WriteDeadAnnotations` - file I/O~~ (removed - added complexity with little value) -- Direct mutation of result data structures - -**Impact**: Can't get analysis results as data. Can't test without capturing I/O. Can't reuse analysis logic for different output formats. - -### P6: Binding/reporting state -**Problem**: `DeadCommon.Current.bindings`, `lastBinding`, `maxValuePosEnd` are per-file state stored globally. - -**Status**: ✅ ALREADY FIXED in previous work - now explicit state threaded through traversals. - ---- - -## End State - -```ocaml -(* ===== IMMUTABLE DATA TYPES ===== *) - -(* Configuration: immutable *) -type config = { ... } - -(* Per-file data - IMMUTABLE, returned by AST processing *) -type file_data = { - source_path : string; - module_name : Name.t; - is_interface : bool; - source_annotations : AnnotationMap.t; (* immutable map *) - decls : DeclMap.t; (* immutable map *) - value_refs : RefMap.t; (* immutable map *) - type_refs : RefMap.t; - file_deps : StringSet.t; (* files this depends on *) -} - -(* Project-wide merged view - IMMUTABLE *) -type merged_view = { - all_annotations : AnnotationMap.t; - all_decls : DeclMap.t; - all_value_refs : RefMap.t; - all_type_refs : RefMap.t; - file_graph : FileGraph.t; -} - -(* Analysis results - IMMUTABLE *) -type analysis_result = { - dead_decls : decl list; - issues : issue list; - annotations_to_write : (string * line_annotation list) list; -} - -(* ===== PHASE 1: AST PROCESSING (local mutable OK) ===== *) - -(* Uses local mutable hashtables for performance, returns immutable *) -let process_file config cmt_infos : file_data = - (* Local mutable state - not visible outside this function *) - let annotations = Hashtbl.create 64 in - let decls = Hashtbl.create 256 in - let refs = Hashtbl.create 256 in - - (* Traverse AST, populate local tables *) - traverse_ast ~annotations ~decls ~refs cmt_infos; - - (* Freeze into immutable data *) - { - source_annotations = AnnotationMap.of_hashtbl annotations; - decls = DeclMap.of_hashtbl decls; - value_refs = RefMap.of_hashtbl refs; - ... - } - -(* ===== PHASE 2: MERGE (pure, parallelizable) ===== *) - -let merge_files (files : file_data StringMap.t) : merged_view = - (* Pure merge of immutable data - can parallelize *) - ... - -(* ===== PHASE 3: ANALYSIS (pure, parallelizable) ===== *) - -let solve_deadness config (view : merged_view) : analysis_result = - (* Pure computation on immutable data *) - (* Can be parallelized, reordered, memoized *) - ... - -(* ===== ORCHESTRATION ===== *) - -let run_analysis ~config ~cmt_files = - (* Phase 1: Process files (can parallelize per-file) *) - let files = - cmt_files - |> List.map (fun path -> (path, process_file config (load_cmt path))) - |> StringMap.of_list - in - (* Phase 2: Merge *) - let merged = merge_files files in - (* Phase 3: Analyze *) - let result = solve_deadness config merged in - (* Phase 4: Report (side effects) *) - report result - -(* Incremental: only re-process changed file *) -let update_file ~config ~files ~changed_file = - let new_data = process_file config (load_cmt changed_file) in - let files = StringMap.add changed_file new_data files in - let merged = merge_files files in - solve_deadness config merged -``` - ---- - -## Refactor Tasks - -Each task should: -- ✅ Fix a real problem listed above -- ✅ Leave the code in a measurably better state -- ✅ Be testable (behavior preserved, but architecture improved) -- ❌ NOT add scaffolding that isn't immediately used - -### Task 1: Remove global "current file" context (P1) - -**Value**: Makes it possible to process files concurrently or out of order. - -**Changes**: -- [x] Create `DeadCommon.FileContext.t` type with `source_path`, `module_name`, `is_interface` fields -- [x] Thread through `DeadCode.processCmt`, `DeadValue`, `DeadType`, `DeadCommon.addDeclaration_` -- [x] Thread through `Exception.processCmt`, `Arnold.processCmt` -- [x] Remove all reads of `Common.currentSrc`, `currentModule`, `currentModuleName` from DCE code -- [x] Delete the globals `currentSrc`, `currentModule`, `currentModuleName` from `Common.ml` - -**Status**: Complete ✅ - -**Test**: Run analysis on same files but vary the order - should get identical results. - -**Estimated effort**: Medium (touches ~10 functions, mostly mechanical) - -### Task 2: Extract configuration into explicit value (P4) - -**Value**: Can run analysis with different configs without mutating globals. Can test with different configs. - -**Changes**: -- [x] ~~Use the `DceConfig.t` already created, thread it through DCE analysis functions~~ -- [x] ~~Replace all DCE code's `!Common.Cli.debug`, `runConfig.transitive`, etc. reads with `config.debug`, `config.run.transitive`~~ -- [x] ~~Make all config parameters required (not optional) - no `config option` anywhere~~ -- [x] Thread config through Exception and Arnold analyses (no `DceConfig.current()` in analysis code) -- [x] Single entry point: only the CLI/entry wrappers (`runAnalysisAndReport`, `DceCommand`) call `DceConfig.current()` once, then pass explicit config everywhere - -**Status**: Complete ✅ (DCE + Exception + Arnold). - -**Test**: Create two configs with different settings, run analysis with each - should respect the config, not read globals. - -**Estimated effort**: Medium (done) - -### Task 3: Source annotations use map → list → merge pattern (P3) - -**Value**: Demonstrates the "local mutable → immutable" architecture for one data type. -Shows the reusable pattern: **map** (per-file) → **list** → **merge** → **immutable result**. - -**Changes**: -- [x] Create `FileAnnotations` module with two types: - - `builder` - mutable, for AST processing - - `t` - immutable, for solver (read-only) -- [x] `DceFileProcessing.process_cmt_file` returns `builder` (local mutable state) -- [x] `processCmtFiles` collects builders into a list (order doesn't matter) -- [x] `FileAnnotations.merge_all : builder list -> t` combines all into immutable result -- [x] Solver receives `t` (read-only, no mutation functions available) -- [x] **Remove solver mutation**: `resolveRecursiveRefs` no longer calls `annotate_dead` -- [x] **Use `decl.resolvedDead` directly**: Already-resolved decls use their stored result - -**Status**: Complete ✅ - -**The Pattern** (reusable for Tasks 4-7): -```ocaml -(* Two types: mutable builder, immutable result *) -type builder (* mutable - for AST processing *) -type t (* immutable - for solver *) - -(* Builder API *) -val create_builder : unit -> builder -val annotate_* : builder -> ... -> unit - -(* Merge: list of builders → immutable result *) -val merge_all : builder list -> t - -(* Read-only API for t *) -val is_annotated_* : t -> ... -> bool -``` - -**Architecture achieved**: -``` -┌─────────────────────────────────────────────────────────────┐ -│ MAP: process each file (parallelizable) │ -│ process_cmt_file → builder (local mutable) │ -└─────────────────────────────────────────────────────────────┘ - │ - ▼ - [ builder list ] - (order doesn't matter) - │ - ▼ -┌─────────────────────────────────────────────────────────────┐ -│ MERGE: combine all (pure) │ -│ merge_all builders → t (immutable) │ -└─────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────┐ -│ ANALYZE: use immutable data │ -│ reportDead ~annotations:t (read-only) │ -└─────────────────────────────────────────────────────────────┘ -``` - -**Key properties**: -- **Order independence**: builders collected in any order → same result -- **Parallelizable**: map phase can run concurrently -- **Incremental**: replace one builder in list, re-merge -- **Type-safe**: `t` has no mutation functions in API - -**Test**: Process files in different orders - results should be identical. - -**Estimated effort**: Small (well-scoped module) - -### Task 4: Declarations use map → list → merge pattern (P2) - -**Value**: Declarations become immutable after AST processing. Enables parallelizable analysis. - -**Pattern**: Same as Task 3 - `builder` (mutable) → `builder list` → `merge_all` → `t` (immutable) - -**Changes**: -- [x] Create `Declarations` module with `builder` and `t` types -- [x] `process_cmt_file` returns `DceFileProcessing.file_data` containing both `annotations` and `decls` builders -- [x] `processCmtFiles` collects into `file_data list` -- [x] `Declarations.merge_all : builder list -> t` -- [x] Solver uses immutable `Declarations.t` -- [x] Delete global `DeadCommon.decls` -- [x] Update `DeadOptionalArgs.forceDelayedItems` to take `~decls:Declarations.t` - -**Status**: Complete ✅ - -**Test**: Process files in different orders - results should be identical. - -**Estimated effort**: Medium (core data structure, many call sites) - -### Task 5: References use map → list → merge pattern (P2) - -**Value**: References become immutable after AST processing. - -**Pattern**: Same as Task 3/4. - -**Changes**: -- [x] Create `References` module with `builder` and `t` types -- [x] Thread `~refs:References.builder` through `addValueReference`, `addTypeReference` -- [x] `process_cmt_file` returns `References.builder` in `file_data` -- [x] Merge refs into builder, process delayed items, then freeze -- [x] Solver uses `References.t` via `find_value_refs` and `find_type_refs` -- [x] Delete global `ValueReferences.table` and `TypeReferences.table` - -**Status**: Complete ✅ - -**Test**: Process files in different orders - results should be identical. - -**Estimated effort**: Medium (similar to Task 4) - -### Task 6: Cross-file items use map → list → merge pattern (P3) - -**Value**: No global queues. Cross-file items are per-file immutable data. - -**Pattern**: Same as Task 3/4/5. - -**Changes**: -- [x] Create `CrossFileItems` module with `builder` and `t` types -- [x] Thread `~cross_file:CrossFileItems.builder` through AST processing -- [x] `process_cmt_file` returns `CrossFileItems.builder` in `file_data` -- [x] `CrossFileItems.merge_all : builder list -> t` -- [x] `process_exception_refs` and `process_optional_args` are pure functions on merged `t` -- [x] Delete global `delayedItems` refs from `DeadException` and `DeadOptionalArgs` - -**Status**: Complete ✅ - -**Note**: `DeadType.TypeDependencies` was already per-file (processed within `process_cmt_file`), -so it didn't need to be included. - -**Key insight**: Cross-file items are references that span file boundaries. -They should follow the same pattern as everything else. - -**Test**: Process files in different orders - results should be identical. - -**Estimated effort**: Medium (3 modules) - -### Task 7: File dependencies use map → list → merge pattern (P2 + P3) - -**Value**: File graph built from immutable per-file data. - -**Pattern**: Same as Task 3/4/5/6. - -**Changes**: -- [x] Create `FileDeps` module with `builder` and `t` types -- [x] `process_cmt_file` returns `FileDeps.builder` -- [x] `FileDeps.merge_all : builder list -> t` -- [x] Thread `~file_deps` through `addValueReference` -- [x] `iter_files_from_roots_to_leaves : t -> (string -> unit) -> unit` (pure function) -- [x] Delete global `FileReferences` from `Common.ml` - -**Status**: Complete ✅ - -**Test**: Build file graph, verify topological ordering is correct. - -**Estimated effort**: Medium (cross-file logic, but well-contained) - -### Task 8: Analysis phase is pure (P5) - -**Value**: Analysis phase works on immutable merged data, returns immutable results. -Can be parallelized, memoized, reordered. - -**Architecture goal**: -``` -merged_view (immutable) - │ - ▼ -solve_deadness (pure function) - │ - ▼ -analysis_result (immutable) - │ - ▼ -report (side effects here only) -``` - -**Approach**: Break into small, behavior-preserving steps. Each step can be verified -before moving to the next. The key is: change return type, then immediately log at -the call site, so behavior stays identical. - ---- - -#### Task 8.1: Create `AnalysisResult` module ✅ - -**Changes**: -- [x] Create `AnalysisResult.ml/mli` with `type t = { issues: Common.issue list }` -- [x] Add constructors: `empty`, `add_issue`, `get_issues` -- [x] Add issue constructors: `make_dead_issue`, `make_dead_module_issue` - -**Verify**: Build succeeds. No behavior change (module not used yet). - ---- - -#### Task 8.2: Make `emitWarning` return issue (behavior preserving) ✅ - -**Changes**: -- [x] Add `makeDeadIssue` (pure function) -- [x] `emitWarning` uses `makeDeadIssue` internally (later removed) - -**Verify**: `make test-analysis` passes with identical output. - ---- - -#### Task 8.3: Make `Decl.report` return issue option (behavior preserving) ✅ - -**Changes**: -- [x] Change `Decl.report` signature to return `issue option` -- [x] Use `makeDeadIssue` internally -- [x] At call site in `reportDead`, log returned issue - -**Verify**: `make test-analysis` passes with identical output. - ---- - -#### Task 8.4: Make `DeadOptionalArgs.check` return issues (behavior preserving) ✅ - -**Changes**: -- [x] Add `foldUnused`, `foldAlwaysUsed` to `OptionalArgs` module -- [x] Change `check` signature to return `issue list` instead of `unit` -- [x] At call site in `resolveRecursiveRefs`, immediately log returned issues - -**Verify**: `make test-analysis` passes with identical output. - ---- - -#### Task 8.5: Collect incorrect annotation issues (behavior preserving) ✅ - -**Changes**: -- [x] Use `makeDeadIssue` for incorrect `@dead` annotation issues -- [x] Log immediately at call site -- [x] Remove `emitWarning` function (no longer needed) - -**Verify**: `make test-analysis` passes with identical output. - ---- - -#### Task 8.6: Make `DeadModules.checkModuleDead` return issue (behavior preserving) ✅ - -**Changes**: -- [x] Change `DeadModules.checkModuleDead` to return `issue option` -- [x] At call sites, log returned issue immediately - -**Verify**: `make test-analysis` passes with identical output. - ---- - -#### Task 8.7: Collect all issues in `reportDead` (behavior preserving) ✅ - -**Changes**: -- [x] Change `Decl.report` to return `issue list` (includes dead module issues) -- [x] Use `List.concat_map` to collect all issues -- [x] Log all issues at the end of `reportDead` - -**Verify**: `make test-analysis` passes with identical output. - ---- - -#### Task 8.8: Return `AnalysisResult.t` from `reportDead` ✅ - -**Changes**: -- [x] Change `reportDead` to return `AnalysisResult.t` instead of `unit` -- [x] Move logging from `reportDead` to caller in `Reanalyze.ml` - -**Verify**: `make test-analysis` passes with identical output. - ---- - -**Status**: Complete ✅ - -**Key guarantee**: The analysis phase (`reportDead`) now returns an immutable -`AnalysisResult.t` containing all dead code issues. Side effects (logging) -only happen in the caller (`Reanalyze.runAnalysis`). - -**Note**: Optional args and incorrect annotation issues were logged inline -during `resolveRecursiveRefs`. Fixed in Task 8b. - -### Task 8b: Collect all issues in AnalysisResult.t (P5) ✅ - -**Value**: Complete the pure analysis phase - all issues returned in result, no inline logging. - -**Problem**: `resolveRecursiveRefs` was logging two types of issues inline: -1. Optional args issues (from `checkOptionalArgFn`) -2. Incorrect `@dead` annotation issues - -These bypassed `AnalysisResult.t` and were logged directly via `Log_.warning`. - -**Changes**: -- [x] Pass `~issues:(Common.issue list ref)` through `resolveRecursiveRefs` -- [x] Collect optional args issues instead of logging inline -- [x] Collect incorrect annotation issues instead of logging inline -- [x] Add collected issues to `AnalysisResult.t` in `reportDead` -- [x] Remove all `Log_.warning` calls from `resolveRecursiveRefs` - -**Status**: Complete ✅ - -**Key guarantee**: No `Log_.warning` calls in `resolveRecursiveRefs`. All issues -are collected in `AnalysisResult.t` and logged by the caller. - -### Task 9: ~~Separate annotation computation from file writing (P5)~~ REMOVED - -**Status**: Removed ✅ - `WriteDeadAnnotations` feature was deleted entirely. - -The `-write` flag that auto-inserted `@dead` annotations into source files was removed -as it added significant complexity (global state, file I/O during analysis, extra types) -for a rarely-used feature. Users who want to suppress dead code warnings can manually -add `@dead` annotations. - -### Task 10: Verify zero `DceConfig.current()` calls in analysis code - -**Value**: Enforce purity - no hidden global reads. - -**Changes**: -- [x] Verify `DceConfig.current()` only called in entry wrappers (CLI / `runAnalysisAndReport`) -- [x] Verify no calls to `DceConfig.current()` in `Dead*.ml`, `Exception.ml`, `Arnold.ml` analysis code -- [x] All analysis functions take explicit `~config` parameter - -**Test**: `grep -r "DceConfig.current" analysis/reanalyze/src/{Dead,Exception,Arnold}.ml` returns zero results. ✅ - -**Estimated effort**: Trivial (done) - -### Task 11: Integration and order-independence verification - -**Value**: Verify the refactor achieved its goals. - -**Changes**: -- [x] Write property test: process files in random orders, verify identical results - - Added `-test-shuffle` CLI flag to randomize file processing order - - Added `test-order-independence.sh` script that runs 3 shuffled iterations and compares output - - Run via `make test-reanalyze-order-independence` (not part of default test) -- [x] Solver takes explicit inputs (no global state) - verified by architecture -- [x] Document the new architecture and API - added "Architecture Diagram" section - -**Test**: The tests are the task. - -**Status**: Complete ✅ - -**Estimated effort**: Small (mostly writing tests) - ---- - -## Execution Strategy - -**Completed**: Task 1 ✅, Task 2 ✅, Task 3 ✅, Task 10 ✅, Task 11 ✅ - -**Remaining order**: 4 → 5 → 6 → 7 → 8 → 9 → 11 (test) - -**Why this order?** -- Tasks 1-2 remove implicit dependencies (file context, config) - ✅ DONE -- Task 3 makes source annotations read-only (solver no longer mutates) - ✅ DONE -- Tasks 4-7 make state **per-file** for incremental updates -- Task 8 makes reporting **pure** with immutable results -- Task 9 separates annotation computation from file writing -- Task 10 verifies no global config reads remain - ✅ DONE -- Task 11 validates everything including incremental updates - -**Key architectural milestones**: -1. **After Task 7**: All state is per-file, keyed by filename -2. **After Task 8**: Solver is pure, returns immutable results -3. **After Task 11**: Incremental updates verified working - -**Time estimate**: -- Best case (everything goes smoothly): 2-3 days -- Realistic (with bugs/complications): 1 week -- Worst case (major architectural issues): 2 weeks - ---- - -## Optional Future Tasks - -### Optional Task: Make OptionalArgs tracking immutable ✅ - -**Value**: `OptionalArgs.t` is now fully immutable. No mutation of declarations. - -**Changes made**: -- [x] Made `OptionalArgs.t` immutable (no mutable fields) -- [x] Added pure functions: `apply_call`, `combine_pair` -- [x] Created `OptionalArgsState` module in `Common.ml` for state map -- [x] `compute_optional_args_state` returns immutable state map -- [x] `DeadOptionalArgs.check` looks up state from map - -**Architecture**: -- Declaration's `optionalArgs` = initial state (what args exist) -- `OptionalArgsState.t` = computed state (after all calls/combines) -- Solver uses `OptionalArgsState.find_opt` to get final state - -**Status**: Complete ✅ - ---- - -## Success Criteria - -After all tasks: - -✅ **Local mutable → Immutable boundary** -- AST processing uses local mutable state (performance) -- Returns **immutable** `file_data` -- Analysis phase works **only** on immutable data - -✅ **Pure analysis phase** -- `solve_deadness : merged_view -> analysis_result` is pure -- No side effects (logging, I/O) in analysis -- Can parallelize, memoize, reorder - -✅ **Incremental updates** -- Replace one file's `file_data` without touching others -- Re-merge is pure function on immutable data -- Re-analyze is pure function on immutable data - -✅ **Order independence** -- Processing files in any order → identical `file_data` -- Merging in any order → identical `merged_view` -- Property test verifies this - -✅ **Static guarantees** -- Type system enforces immutability after AST processing -- No `ref` or mutable `Hashtbl` visible in analysis phase API -- Compiler catches violations - -✅ **Testable** -- Test AST processing in isolation (per-file) -- Test merge function in isolation (pure) -- Test analysis in isolation (pure) -- No mocking needed - just pass immutable data diff --git a/analysis/reanalyze/src/dead_value.ml b/analysis/reanalyze/src/dead_value.ml index f6da7f9d095..9ded2f36e48 100644 --- a/analysis/reanalyze/src/dead_value.ml +++ b/analysis/reanalyze/src/dead_value.ml @@ -159,7 +159,7 @@ let rec collect_expr ~config ~refs ~file_deps ~cross_file ~direct_callees (* [Location.none] identifies a top-level [Tstr_eval], which executes unconditionally, so use the expression's own position. Both liveness implementations deliberately treat non-declaration - positions as live; that behavior is load-bearing here. *) + positions as live; this analysis depends on that behavior. *) if binding = Location.none then loc_from.loc_start else binding.loc_start in diff --git a/compiler/core/FIXME.adoc b/compiler/core/FIXME.adoc deleted file mode 100644 index ce60151e00b..00000000000 --- a/compiler/core/FIXME.adoc +++ /dev/null @@ -1,79 +0,0 @@ - - -# alias table is not reliable - -mario_game.03flattern.lam - -[source] ------------------ - with (304 t1/1824 s1/1825 o1/1826 t2/1827 o2/1829) - (let (match/2332 =a t1/1824 match/2333 =a t2/1827) - (catch - (catch - (catch - (catch (if (>= match/2332 3) (exit 291) (exit 290)) - with (291) - (if (isint match/2333) - (if (!=[int] match/2333 1) (exit 290) (exit 288)) - (let (typ/2330 =a (field 0 match/2333)) - (exit 289 typ/2330)))) - with (290) - (seq - (apply (field 17 Object/1601) o1/1826 t1/1824 - s1/1825) - [0: 0a 0a])) - with (288) - (seq (apply (field 16 Object/1601) o2/1829) - (apply (field 18 Object/1601) o1/1826) [0: 0a 0a])) - with (289 typ/1867) - (let - (updated_block/1869 = - (apply (field 15 Object/1601) o2/1829 context/1786) - spawned_item/1870 = - (apply (field 20 Object/1601) (field 6 o1/1826) - o2/1829 typ/1867 context/1786)) - (seq - (apply (field 17 Object/1601) o1/1826 t1/1824 - s1/1825) - (makeblock 0 (makeblock 0 updated_block/1869) - (makeblock 0 spawned_item/1870))))))) ------------------ - -Here [match/2332] is aliased [t/1824] - -mario_game.04.simplify_exits.lam - -[source] ------------------- - (if (>= match/2379 2) - (let - (t1/2571 =a t1/1817 - s1/2572 =a s1/1818 - o1/2573 =a o1/1819 - t2/2574 =a t2/2349 - o2/2575 =a o2/2350 - match/2332 =a t1/2571 - match/2333 =a t2/2574) - (catch - (if (>= match/2332 3) - (if (isint match/2333) - (if (!=[int] match/2333 1) (exit 290) - (seq - (apply (field 16 Object/1601) - o2/2575) - (apply (field 18 Object/1601) - o1/2573) - [0: 0a 0a])) - (let - ------------------- - -Here [match/2332] is aliased [t/1824] - -Here the exit code is inlined, [t/1824] is renamed into [t1/2571] -(function inlining and renaming), [match/2332] now should be aliased -to [t1/2571], but since the alias table is not updated, it still point -to the old one which result in wrong optimizations. - -The reaon is that when we do the inlining we refresh the parameters, -but forgot update the alias table, so the aliases are stale diff --git a/compiler/core/README.md b/compiler/core/README.md new file mode 100644 index 00000000000..28ef3609bf6 --- /dev/null +++ b/compiler/core/README.md @@ -0,0 +1,69 @@ +# Lambda, Lam, and JavaScript generation + +This directory contains the compiler backend after typedtree translation. It +owns ReScript's Lam representation, Lam optimization passes, JavaScript IR, +and JavaScript output. + +## Pipeline and code map + +Typedtree translation in `compiler/ml/translcore.ml` and +`compiler/ml/translmod.ml` produces the `Lambda` representation defined in +`compiler/ml/lambda.mli`. + +[`lam_convert.ml`](lam_convert.ml) +: Converts `Lambda.lambda` to the ReScript-specific [`Lam.t`](lam.mli), + normalizes aliases, and collects potential module dependencies. + +`lam_pass_*.ml` and the other `lam_*.ml` modules +: Analyze and transform Lam. [`lam_compile_main.ml`](lam_compile_main.ml) + coordinates the backend pass sequence; read it before inserting or + reordering a pass. + +[`lam_compile.ml`](lam_compile.ml) +: Lowers Lam to JavaScript IR. Primitive-specific and FFI lowering is split + into `lam_compile_primitive.ml`, `lam_compile_external_call.ml`, and related + modules. + +[`j.ml`](j.ml) +: Defines JavaScript expressions, statements, and blocks. + +`js_pass_*.ml` and other `js_*.ml` modules +: Analyze and transform JavaScript IR. `js_dump*.ml` renders the final program, + while [`js_implementation.ml`](js_implementation.ml) coordinates compilation + from a source file. + +## Changing a representation + +`Lambda` and `Lam` have similarly named constructors but are distinct IRs. +When adding or changing one, search every producer, traversal, optimizer, +printer, serializer, and consumer of that specific type. Do not assume a match +on the other representation covers it. + +Check persistence boundaries as part of the change. `Lam.t` can be stored in +`.cmj` data through `js_cmj_format`; a constructor or payload change therefore +changes cached compiler data even when generated JavaScript is unchanged. + +Keep representation contracts in the owning `.mli`, pass-order or analysis +invariants beside the pass implementation, and this guide limited to +navigation. Remove completed design notes rather than retaining them as an +alternative description of the current backend. + +## Testing and inspection + +Run the full compiler tests for backend changes: + +```sh +make test +``` + +End-to-end fixtures under `tests/tests/` check generated `.mjs` output. Add +focused OUnit coverage for an isolated analysis or transformation. Use the +compiler flags below to compare intermediate forms for a small source file: + +```sh +./cli/bsc.js -dtypedtree example.res +./cli/bsc.js -drawlambda example.res +``` + +For Lam-specific debugging, use [`lam_print.ml`](lam_print.ml) at the relevant +pass boundary and remove temporary output before committing. diff --git a/compiler/core/design.md b/compiler/core/design.md deleted file mode 100644 index 6b319d18227..00000000000 --- a/compiler/core/design.md +++ /dev/null @@ -1,462 +0,0 @@ - -# Ideas aobut boolean support -## The cases when boolean representation is not transparent - -- printing - -```ocaml -Console.log true -``` - -- pattern match - -```ocaml -let f x y = - match x,y with - | true, false -> 0 -``` - -- comparison - -```ocaml -if v = true then -``` - -## Where JS boolean could be introduced - -JS operatons which could generate JS booleans -- `not` -- Equality comparison - -# `and`, `or` is fine -In JS, `and`, `or` is untyped, but it is a superset of OCaml semantics: - - -```js -x && y -/* equivalent to */ -x ? x : y -``` - - - -There is no coersion so `1&&0`, `1&&1`, `0&&1`, `0&&0` are all the same as JS version. -Same for `or` - -but `not` is not the same as `!`, `!` will do the conversion to enforce its result is JS boolean - -**ATT** if you want to support JS boolean better, think twice, it is -really hard to give two booleans first class support, and such bugs -are very hard to find (since in most cases they behave the same), so -what we can do is produce OCaml bool exclusively, only pass JS boolean -to JS ffi which requires it exclusively (very rare) -**NOTE** since OCaml boolean is everywhere, while JS boolean only happens in the FFI, we should by default produce OCaml boolean in the IR, and mark JS boolean explicitly instead. - -## It does not affect `if_then_else` compilation - -since JS if is more capable, there is no need do any coercion - - -# Arity handling - -The runtime support is in `Curry` module, we have several functions - -## `Curry.app` - -```ocaml -Curry.app f args -(** [f] is an curried function, [args] are supplied arguments - if matches then like normal function apply. - if over-supply, take the first [arity] arguments, - fully apply and continue [app f' rest] - if under-supply, - create a closure, wait until all arguments ready. - [Note, we don't necessary - need a closure here, we can have some - data structures like bytecode] - *) -``` - -```ocaml - -Curry.curry_N o a1 a2 .. arity -(** used by [Curry._N] *) - -Curry._N o a1 a2 .. aN -(** - A fast version of [app f [|a1; a2; ..; aN |]]. - Check the arity of [o]if it hits, just do the application -*) - -Curry.__N o -(** - Make sure the output of [o] is arity [N]. - This is used to convert a curried function [o] into - uncurried. for example - {[ - fun [@bs] x y -> f x y - ]} - Another use case: - {[ - external f : ('a -> 'b [@uncurry]) -> unit - - f g (* The compiler will do such converison internally*) - ]} - Guess the arity of [o], if it hit, then return [o] - - Note when we want to target arity 0, in the first case - {[ - fun [@bs] () -> f () - ]} will be compiled as - {{ - fun () -> f (0) - }} - SO [Curry.__0] will not be triggered - - We also have - some special logic to when converted to arity 0 - in external settings -*) - -``` - -# Toplevel module exports - -Global exports identifiers are extracted from `Translmod.get_export_identifiers` -instead of inferred from lambda expression or cmi file. - -- We need be careful about externals. -- Reading from fresh generated cmi is expensive. - -# Variable usage - -Lalias-bound variables are never assigned, so it can only -appear in `Lvar`, then it is easy to eliminate it - - -# interaction between `variadic` and `|>` - -Note in general, it is fine whether we do beta reduction or not, it is just optimization. - -However, since we introduced `variadic` which does require the `spliced argument` to be captured - -```ocaml -spliced_external a0 a1 [|b0;b1|] -``` - -There are two cases where get things complicated, people don't think `|>` is a function - -```ocaml -x |> spliced_external a0 a1 [|b0;b1|] -``` -Even though `|>` is a function, and `spliced_external` is escaped here, but people would -expect it is equivalent to - -```ocaml -spliced_external a0 a1 [|b0;b1|] x -``` - -So our optimizer needs to handle this case to make sure `spliced_external` not escaped, -also becaues the interaction of `[@variadic]` and `[@send]`, the spliced argument -is no longer in tail position, so that people can write such code - -```ocaml -spliced_external a0 a1 [|b0;b1|] -``` -Internally in lambda layer it would be - -```ocaml -(fun c0 c1 c2 c3-> spliced_external c0 c1 c2 c3) a0 a1 [|b0;b1|] -``` - -We can simply do inlining, it may have side efffect in `b0`, `b1`, our optimizer also need handle such case. - -Maybe in the future, we should lift the restriction about `variadic` (delegate to `slow` mode when we can not resolve it statically, my personal expereince is that people will complain about why it fails to compile more than why it is slow in some corner cases) - -Note when we pattern match over the original lamba,`Levent` needs to be removed as early as possible. Due to the existence of `Levent`, we can not pattern match over nested original raw lambda. - -We turned off event generation temporarily - -# safe way to test undefined - -Note such logic is already wrong: - -```js -var x = undefined_value // thrown here -if (typeof x === "undefined"){ - ... -} -``` - - -# `#` primitive handling - -1. Some primitives introduced are for performance reasons, for example: - -`#String.fromCharCode` which is essentialy the same as - -```ocaml -external of_char : char -> string = "String.fromCharCode" -[@@val] -``` - -We introduced `#` so that we can do some optimizations. - -2. Some of them are not expressible in OCaml FFI, for example -'#is_instance_array', -'#gt' - -3. Some of them require a runtime polyfill support - - -# runtime - -## anything to string -http://www.2ality.com/2012/03/converting-to-string.html -Note that `""+ Symbol()` does not work any more, we should favor `String` instead - - -# Name mangling - -## let bound identifier mangling - -### keyword -Note there are two issues, if it is keyword, the output may not be parsable if we don't do name mangling - - -```js -> var case = 3 -< SyntaxError: Cannot use the keyword 'case' as a variable name. -``` -### global variable -If it is global variable, it is parsable, it may trigger even subtle errors: - -```js -(function(){ 'use strict'; var document = 3; console.log(document)})() -VM1146:1 3 -3 -``` -This could be _problematic_ for bindings -```ocaml -let process = 3 -Process.env##OCAML -``` -In general global variables would be problematic for bindings - -## property name mangling - -Nowadays, JS engine support keywords as property name very well - -```js -var f = { true : true, false : false } -``` - -But it has problems when it is too simple for parsing -```js -var f = { true, false} // parsign rules ambiguity -``` - -If we don't do ES6, we should not go with name mangling, however, it is mostly due to we can -not express these keywords, such as `_open` as property in OCaml, so we did the name mangling - -# Curry/Tuple: two kinds of function - -OCaml indeed support two kind calling convention. - -```ocaml -(Lfunction (Tupled(a0,a1,a2))) [a0,a1,a2] -``` - -and - -```ocaml -(Lfunction Curried (a0,a1,a2)) a0 a1 a2 -``` - - -They also affect how beta reduction works, - -```ocaml -| Lapply(Lfunction(Curried, params, body), args, _) - when optimize && List.length params = List.length args -> - count bv (beta_reduce params body args) -| Lapply(Lfunction(Tupled, params, body), [Lprim(Pmakeblock _, args,_)], _) - when optimize && List.length params = List.length args -> - count bv (beta_reduce params body args) -``` - -It is generated by the backend via an argument -`untuplify_fn` in `transl_function`. -Note if we want to take advantage of it in the future we need -translate ` f x` into (when `f` is `fun (a0,a1) -> a0 + a1 ` - -```ocaml -f (x[0],x[1]) -``` - -currently it is turned on only in native mode - -```ocaml -and transl_function loc untuplify_fn .. -``` - -two call sites - -```ocaml -transl_function exp.exp_loc false ... -transl_function e.exp_loc !Clflags.native_code ... -``` - -# JS exception wrap and unwrap - -### Pack and Unpack OCaml exceptions - -http://eli.thegreenplace.net/2013/10/22/classical-inheritance-in-javascript-es5 - -http://stackoverflow.com/questions/1382107/whats-a-good-way-to-extend-error-in-javascript - - -https://caml.inria.fr/mantis/print_bug_page.php?bug_id=7375 - -http://stackoverflow.com/questions/3734236/how-can-i-rethrow-an-exception-in-javascript-but-preserve-the-stack -```js -class OCamlError extends Error{ - constructor(payload){ - super("OCamlError") - this.payload = payload - } -} -``` - -We can see the output from typescript to get a sense of what it will be transpiled into. - -This works reasonably well (tested on Safari and Chrome) - -```js -function OCamlError(camlExnData){ - var self = Error.call(this, "OCamlError") - self.camlExnData = camlExnData - return self -} -``` - -```js -function unpackError(exn){ - if(exn.camlExnData !== undefined){ - return exn.camlExnData - } else { - return exn - } -} -``` -```js -function packError(exn){ - if (Obj.tag(exn) === 248){ - return new OCamlError(exn) - } else { - return exn - } -} -``` -So whenever we raise an OCaml exception, we always wrapped it as a JS exception. -Now we unpack it, it could be OCaml exception or JS exception, so we did -a runtime dispatch. - -Some potential optimization - -```ocaml -try f x with -Not_found -> .. -JsExn e -``` - -currently it would be transalted as - -```js -try { - f(x) - } -catch(e){ - var e = unpackError(e) - if (e === "Not_found"){ - ... - } else { - throw packError(e) // re-raise - } -} -``` - -need check `Praise of raise_kind` - -*Conclusion*: it is very hard to get it right when changig ocaml exception representation and js exception representation at the same time in the combination of *re-raiase* - - -# Several module components can have the same name #978 - -Note: ReScript compiler complains if exports have the same component name, this keeps its soundness. - -A funny thing is that open variants does not have record disambiguion so that for: - -```ocaml -type a = .. -type b = .. -type a += A -type b += A -``` - -The compiler will only expose the last `A`, which means, ReScript will not complain, the limitation -of the compiler preserves its soundness - -# Print import module names - -for `create_js_module`, we first create a mapping to make it a proper -module name, (also cached in a hashtbl). Note it is not a Js id, which -fails `Ext_ident.is_js` - -# compilation - -# static catches - -# Comparison semantics - -Cases when commparison are specialized (Note we need make sure the specialized version -is consistent with the generalized version): - -- caml_int_max/min -- caml_bool_max/min -- caml_float_max/min -- caml_string_max/min -- caml_nativeint_max/min -- caml_int32_max/min -- caml_int64_max/min - -- int_equal[null/undefined/nullable] [not] -- bool_equal[null/undefined/nullable] [not] -- float_equal[null/undefined/nullable] [not] -- string_equal[null/undefined/nullable] [not] -- nativeintequal[_null/unefined/nullable] [not] -- int32_equal[_null/undefined/nullable] [not] -- int64_equal[_null/undefined/nullable] [not] - -- int_lessthan[greaterthan] [lessequal] [greaterequal] -- bool_lessthan[greaterthan] [lessequal] [greaterequal] -- float_lessthan[greaterthan] [lessequal] [greaterequal] -- string_lessthan[greaterthan] [lessequal] [greaterequal] -- nativeint_lessthan[greaterthan] [lessequal] [greaterequal] -- int32_lessthan[greaterthan] [lessequal] [greaterequal] -- int64_lessthan[greaterthan] [lessequal] [greaterequal] - -- int_compare -- bool_compare -- float_compare -- string_compare -- nativeint_compare -- int32_comapre -- int64_compare - -So far we haven't specialized option comparison, but we need be careful when -we do the optimizer, e.g, `Js_exp_make.int_comp`, we need make sure the peepwhole is consistent - - - - diff --git a/compiler/core/destruct_exn.md b/compiler/core/destruct_exn.md deleted file mode 100644 index 201c8305ee0..00000000000 --- a/compiler/core/destruct_exn.md +++ /dev/null @@ -1,81 +0,0 @@ - - - -Its essential is - -```ocaml -external destruct : 'b -> (exn -> 'a) -``` - -However it does not prevent things like - -```ocaml -destruct v begin fun exn -> - Console.log exn ; - match exn with - | .. - | .. -``` - -Here it forces us to answer whether `v` is exception or not, - -while such syntax below does not need us answer `v` is exception -or not, it just asks us to answer it matches a branch of exception or not which can be done in a sound way. - -```ocaml -match%exn v with -| .. -| .. -``` - -However, we need make sure such cases not happen - -```ocaml -match%exn v with -| e -> ... - -``` -Or any vagous pattern which needs us to answer if -it is an exception or not - - -Another proposal is -```ocaml -match%exn computation with -| A .. -| B .. -| JsExn .. -| v -> .. -``` - -Here we pack the data `v` - -==> -``` - -match (Primitive_exceptions.internalToException computation) with -| A .. -| B -| exception .. ) -``` - -The same problem is - -``` -match (Primitive_exceptions.internalToException computation) with -| _ -> .. -``` - -What will happen if JS side `raises` an OCaml extensible variant, -we view it as OCaml exception.. -It is slightly different in OCaml, since it always start from `catch(id)..` -where `id` is defined by the compiler - -Another very similar proposal would be - -```ocaml -fun[@bs:exn] e -> - match e with - | JsExn .. - | .. -``` diff --git a/compiler/depends/astdump_main.md b/compiler/depends/astdump_main.md deleted file mode 100644 index f2f7c1e6910..00000000000 --- a/compiler/depends/astdump_main.md +++ /dev/null @@ -1,13 +0,0 @@ -current Ast format (10/10/2020) - --- input_binary_int ic (size) -module, -module, -... ---- seek_in ic (pos_in ic + size) -fname -marshalled ast - - - - diff --git a/compiler/ext/README.md b/compiler/ext/README.md deleted file mode 100644 index b4a1edaeab6..00000000000 --- a/compiler/ext/README.md +++ /dev/null @@ -1 +0,0 @@ -This folder hosts some of the utils we use in ReScript, internally. diff --git a/compiler/ext/encoding.md b/compiler/ext/encoding.md deleted file mode 100644 index 59228c0298b..00000000000 --- a/compiler/ext/encoding.md +++ /dev/null @@ -1,42 +0,0 @@ - - - - - - -```c -CAMLprim value caml_ml_string_length(value s) -{ - mlsize_t temp; - temp = Bosize_val(s) - 1; - Assert (Byte (s, temp - Byte (s, temp)) == 0); - return Val_long(temp - Byte (s, temp)); -} -``` - -Like all heap blocks, strings contain a header defining the size of -the string in machine words. The actual block contents are: -- the characters of the string -- padding bytes to align the block on a word boundary. - The padding is one of - 00 - 00 01 - 00 00 02 - 00 00 00 03 - on a 32-bit machine, and up to 00 00 .... 07 on a 64-bit machine. - -Thus, the string is always zero-terminated, and its length can be -computed as follows: - - number_of_words_in_block * sizeof(word) - last_byte_of_block - 1 - -The null-termination comes handy when passing a string to C, but is -not relied upon to compute the length (in Caml), allowing the string -to contain nulls. - -so, suppose - -"" -> `8 - 7 - 1 ` -"a" -> `8 - 6 - 1` -"0123456" -> `8 - 0 - 1` -"01234567" -> `2 * 8 - 7 - 1` \ No newline at end of file diff --git a/compiler/ml/README.md b/compiler/ml/README.md new file mode 100644 index 00000000000..717e44fcd9c --- /dev/null +++ b/compiler/ml/README.md @@ -0,0 +1,137 @@ +# Type checker + +This directory contains the compiler's type representation, type checker, and +typed tree. It also contains several later frontend passes inherited from the +same compiler layer. This guide is an index to the type-checking code; the +module interfaces and local implementation comments define the detailed +contracts. + +## Where to start + +For a surface-language expression, declaration, or module feature, follow the +parsetree node through these modules: + +- [`parsetree.ml`](parsetree.ml) defines the current untyped AST. + `parsetree0.ml` is the frozen compatibility AST and must not be changed. +- [`typecore.ml`](typecore.ml) checks expressions and patterns. +- [`typetexp.ml`](typetexp.ml) translates source type expressions. +- [`typedecl.ml`](typedecl.ml) checks type declarations. +- [`typemod.ml`](typemod.ml) checks structures, signatures, and modules. Its + `type_structure` entry point coordinates structure typing. +- [`typedtree.ml`](typedtree.ml) defines the result consumed by later compiler + passes and tooling. + +For a change to inference, unification, generalization, subtyping, or type +copying, start with the public contracts in [`ctype.mli`](ctype.mli) and +[`btype.mli`](btype.mli), then read the corresponding implementation around the +operation being changed. These modules operate on shared mutable type graphs; +local-looking mutations can affect aliases and speculative checks. + +For module inclusion and signature compatibility, start with +[`includecore.mli`](includecore.mli), [`includemod.mli`](includemod.mli), and +[`mtype.mli`](mtype.mli). + +## Main data and operations + +[`types.mli`](types.mli) +: Internal type expressions and declarations. A `type_expr` is a mutable graph + node, not an immutable syntax tree. Use the representation functions exposed + by `Btype` before interpreting linkable state. + +[`typedtree.mli`](typedtree.mli) +: Typed expressions, patterns, declarations, and modules. Nodes retain the + environment in which they were checked. + +[`env.mli`](env.mli) +: Typing environments, lookup, persistent signatures, and local constraints. + +[`btype.mli`](btype.mli) +: Operations on the type representation: representatives, graph traversal, + the mutation trail, snapshots, and scoped copying. Read its copy-session + contract before calling `copy_type_desc` directly. + +[`ctype.mli`](ctype.mli) +: Type-checking operations built on that representation: instantiation, + generalization, unification, object-field constraints, enlargement, and + subtyping. + +[`subst.mli`](subst.mli) +: Substitution and copying across environments and persistence boundaries. + `for_saving` has stronger independence requirements than an ordinary copy. + +## Polymorphic value positions + +A `Tpoly` node represents a type scheme, so the operation depends on whether +the surrounding syntax consumes or defines a value at that scheme: + +- An elimination site instantiates the scheme for one use. Object-field reads + do this in `object_field_use_type`. +- An introduction site must show that the expression is at least as general as + the scheme. `type_let` for polymorphic annotations, `type_label_exp` for + record fields, and `type_object_field_value` for object-field assignments all + type the expression at a fixed instance and call `check_univars`. + +The `fixed` argument of `Ctype.instance_poly` controls the copying of fixed +polymorphic-variant rows; it is not by itself an introduction/elimination +marker. The generality check is what distinguishes introduction. After a +successful check, the typed expression carries an ordinary instance rather +than the fixed checking instance. + +Do not copy `type_label_exp`'s retry for expansive expressions into another +introduction site by default. That retry recovers completeness lost through +record-label type propagation and is specific to that typing path. + +## Choosing the right level for documentation + +- Put caller-observable requirements in a module interface. Examples include + whether an operation mutates its input graph, requires a copy session, can + leave deferred constraints, or must be paired with backtracking. +- Put representation invariants and algorithm ordering in the implementation + that owns them. Examples include how copy-session marks are installed and + restored, or why row openness is sampled before row tails are unified. +- Keep comments at call sites when a high-level algorithm combines mechanisms + whose interaction is otherwise easy to miss. Coercion in `typecore.ml`, for + example, has both an enlargement-and-unification path and a subtyping path. +- Put broad navigation and debugging advice here. Do not duplicate detailed + invariants from source comments in this guide. + +When a change introduces a new form of mutable state in a type node, document +at least its semantic states, representative operation, sharing and copying +rules, trail/backtracking behavior, and persistence behavior. Tests should +cover aliasing, instantiation independence, speculative failure, and saving +when those properties apply. + +## Testing and inspection + +From the repository root: + +```sh +make # build the compiler and build system +make test # build the library and run the complete test suite +make checkformat # check formatting +``` + +Useful focused suites include `tests/ounit_tests/` for internal operations and +`tests/build_tests/super_errors/` for type errors. Add end-to-end cases under +`tests/tests/` when a type-system change also affects accepted programs or +generated JavaScript. + +To inspect the compiler pipeline for a small source file: + +```sh +./cli/bsc.js -dparsetree example.res +./cli/bsc.js -dtypedtree example.res +./cli/bsc.js -drawlambda example.res +``` + +Small source probes are useful evidence, but they do not by themselves test +graph sharing, backtracking, or persistence. Add a unit test when the property +cannot be observed reliably through source syntax. + +## Background reading + +The checker uses level-based generalization. Oleg Kiselyov's +[Efficient and Insightful Generalization](https://okmij.org/ftp/ML/generalization.html) +is useful background before changing generalization or instantiation. External +material explains the underlying techniques, but this repository's interfaces, +implementation comments, and tests define current ReScript behavior. diff --git a/compiler/ml/btype.ml b/compiler/ml/btype.ml index 435fdb95acd..b76ba8c0018 100644 --- a/compiler/ml/btype.ml +++ b/compiler/ml/btype.ml @@ -74,6 +74,13 @@ let default_mty = function (**** Definitions for backtracking ****) +(* The backtracking trail owns semantic changes made while checking a + speculative constraint. [log_type] and [set_level] use [last_snapshot] to + avoid logging nodes allocated after the active snapshot. Link compression + and separately allocated references cannot use that node-id test, so their + changes are logged whenever a trail is active. Temporary graph-copy memos + are not trail entries; [type_copy_session] below owns and restores them. *) + type change = | Ctype of type_expr * type_desc | Ccompress of type_expr * type_desc * type_desc @@ -414,6 +421,14 @@ type type_copy_session = { mutable copy_policy_memo: (int, bool) Hashtbl.t option; } +(* A copy session owns temporary writes made to the source graph. [Tsubst] + marks live on type nodes and are visible across the active session stack: a + nested copy which encounters an outer mark reuses that copied node. + Mutability duplication membership is instead per-session; if a nested copy + reaches a cell independently, it creates and owns a further duplicate. + Several copy calls in one session deliberately share both memo kinds. + Session writes are raw writes restored on exit; semantic mutability changes + use the backtracking trail instead. *) let type_copy_sessions = ref [] let begin_type_copy_session () = @@ -439,10 +454,11 @@ let end_type_copy_session () = type_copy_sessions := rest | [] -> assert false -(* Duplicate a mutability cell for the current copy session: the original - representative is temporarily linked to the duplicate, so every field - copied in this session that shares the cell reaches the same duplicate - (the former [dup_kind] idiom); [cleanup_types] restores the originals. *) +(* Duplicate a mutability class for the current copy session. The source + representative temporarily links to the duplicate, so all fields in this + session which shared the source class reach the same duplicate. Semantic + operations follow representatives and therefore affect the copy while the + link is installed; ending the session restores the source cell. *) let dup_mutability r = let session = current_type_copy_session () in let r = mutability_ref_repr r in @@ -504,7 +520,10 @@ let rec copy_type_desc ?(keep_names = false) ?(fresh_mutability = false) f = (generic terminator) duplicates each cell once per session via [dup_mutability], so aliases within the instance stay correlated while the scheme and sibling instances are untouched; other copies hold the - shared representative, so promotions reach every occurrence. *) + shared representative, so promotions reach every occurrence. Determine + this policy before copying either child: recursive copying can install + [Tsubst] marks in the row, and OCaml record-field evaluation order must + not decide which terminator is classified. *) let mutability = if fresh_mutability || row_terminator_generic f_.rest then dup_mutability f_.mutability diff --git a/compiler/ml/btype.mli b/compiler/ml/btype.mli index d8f47a5142b..fbe452e2bc8 100644 --- a/compiler/ml/btype.mli +++ b/compiler/ml/btype.mli @@ -123,15 +123,30 @@ val copy_type_desc : (type_expr -> type_expr) -> type_desc -> type_desc -(* Copy on types *) +(** Copy one type description, using the supplied function to copy children. + + Run this operation inside [with_copy_session]. By default, object-field + mutability classes are duplicated when the field's row ends in a generic + variable and shared otherwise. [fresh_mutability:true] instead gives the + result classes which are independent from the source, while preserving + class sharing among fields copied in the same session. *) val copy_row : (type_expr -> type_expr) -> bool -> row_desc -> bool -> type_expr -> row_desc val save_desc : type_expr -> type_desc -> unit -(* Save a type description *) +(** Record a temporary change to a source type description in the current copy + session. The session restores the description when it ends. *) val with_copy_session : (unit -> 'a) -> 'a +(** Run a graph-copy operation with scoped temporary state. + + The scope is nestable and restores its own source-graph changes on normal + return and on exceptions. Repeated low-level copy calls in one scope share + copy memos. An inner operation can reuse a copied node through an outer + [Tsubst] mark; if it independently duplicates a mutability class, that + duplicate belongs to the inner scope. Each scope restores only changes it + owns. This restoration is separate from the [snapshot]/[backtrack] trail. *) val lowest_level : int (* Marked type: ty.level < lowest_level *) @@ -212,9 +227,10 @@ val set_name : val set_row_field : row_field option ref -> row_field -> unit val set_univar : type_expr option ref -> type_expr -> unit -(* Logged (backtrackable) update of a mutability cell: promotion +(* Logged (backtrackable) update of a terminal mutability cell: promotion ([Mutability_value Mutable]) or an equivalence-class merge - ([Mutability_link]). *) + ([Mutability_link]). Resolve the first argument with + [mutability_ref_repr] before calling. *) val set_mutability : field_mutability ref -> field_mutability -> unit (* Terminal cell of a link chain / its semantic value. *) diff --git a/compiler/ml/ctype.ml b/compiler/ml/ctype.ml index 71cf1b5b3b9..883aa379bdb 100644 --- a/compiler/ml/ctype.ml +++ b/compiler/ml/ctype.ml @@ -699,13 +699,15 @@ let rec find_repr p1 = function | Mlink {contents = rem} -> find_repr p1 rem (* - Generic nodes are duplicated, while non-generic nodes are left - as-is. - During instantiation, the description of a generic node is first - replaced by a link to a stub ([Tsubst (newvar ())]). Once the - copy is made, it replaces the stub. - After instantiation, the description of generic node, which was - stored by [save_desc], must be put back, using [cleanup_types]. + Ordinary instantiation copies generic nodes and reuses non-generic nodes. + Pattern typing can pass [partial] to copy selected non-generic nodes as + fresh variables as well. [env] does not affect that choice; when local GADT + constraints are present, it only records the corresponding fresh instance. + + A copied source node is temporarily marked with [Tsubst]. These marks are + visible to nested copy operations, which is intentional: copies made in the + same operation can preserve graph sharing. [with_copy_session] restores the + marks and the temporary mutability links installed by [copy_type_desc]. *) let abbreviations = ref (ref Mnil) @@ -2293,8 +2295,9 @@ and unify_fields env (ty1 : Types.type_expr) (ty2 : Types.type_expr) = and fields2, rest2 = flatten_fields ty2 in let pairs, miss1, miss2 = associate_fields fields1 fields2 in let l1 = (repr ty1).level and l2 = (repr ty2).level in - (* Row openness before the rests are instantiated below: an [Immutable] - field may be promoted to [Mutable] only while its row is open. *) + (* Sample openness before unifying the row tails below. [unify_mutability] + requires this pre-unification state because tail unification can close an + open row before the matching fields are processed. *) let open1 = is_Tvar (repr rest1) and open2 = is_Tvar (repr rest2) in let va = make_rowvar @@ -2340,6 +2343,9 @@ and unify_fields env (ty1 : Types.type_expr) (ty2 : Types.type_expr) = raise exn and unify_mutability ~open1 ~open2 f1 f2 = + (* [open1] and [open2] describe the rows before their tails were unified. + Promotion and class merging operate on representatives, and are recorded + on the ordinary type-checker trail. *) let r1 = mutability_ref_repr f1.f_mut and r2 = mutability_ref_repr f2.f_mut in if r1 != r2 then ( (match (!r1, !r2) with @@ -2701,7 +2707,9 @@ let filter_object_field_for_write env name ty : | Tobject f -> write_field ~can_promote:(is_Tvar (object_row ty)) f | _ -> Error Owrite_missing -(* Unify [ty] and [{.. name: 'a}]. Return ['a]. *) +(* Require a readable [name] field and return its type. If lookup reaches a + [Tvar] row terminator, it can acquire the missing field. Rigid and closed + rows cannot. This operation does not require or introduce write capability. *) let filter_method env name ty = let ty = expand_head_trace env ty in match ty.desc with @@ -3360,7 +3368,7 @@ let rec build_subtype env visited loops posi level t = in let t1', c = build_subtype env visited loops posi level' t1 in if c > Unchanged then (newty (Tobject t1'), c) else (t, Unchanged) - | Tfield ({typ = t1; rest = t2} as f) (* Always present *) -> + | Tfield ({typ = t1; rest = t2} as f) -> let t1', c1 = match mutability_repr f.mutability with | Asttypes.Mutable -> @@ -3868,10 +3876,13 @@ and subtype_fields env trace ty1 ty2 cstrs = write permission. *) subtype_rec env ((f1.f_typ, f2.f_typ) :: trace) f1.f_typ f2.f_typ cstrs | Mutable -> - (* Writable target: delegate to unification of the two fields. The - source fragment uses [rest1], so [unify_mutability] can promote an - [Immutable] source field only when the source row is open. Field - type unification also enforces equivalence. *) + (* A writable target requires equivalent field types. Add a deferred + unification constraint over transient one-field fragments. The + source fragment retains [rest1], allowing [unify_fields] to observe + whether the source row was open before it unifies the tails and, if + so, promote the field. These fragments borrow the existing cells; + they are consumed by deferred constraint enforcement and must not + be copied, generalized, or persisted. *) let src = newty (Tfield @@ -4116,6 +4127,11 @@ let clear_hash () = Type_hash.clear nondep_variants let with_nondep_copy_session f = + (* [nondep_type_rec] cannot use [Tsubst] because abbreviation expansion can + unify while the copy is in progress. Mutability cells may nevertheless be + temporarily linked by [copy_type_desc]. Semantic operations follow their + representatives, so any such unification updates the copy under + construction; the source link is restored when this session ends. *) with_copy_session (fun () -> match f () with | result -> diff --git a/compiler/ml/ctype.mli b/compiler/ml/ctype.mli index 0d008f72399..a25d6d62e22 100644 --- a/compiler/ml/ctype.mli +++ b/compiler/ml/ctype.mli @@ -144,17 +144,23 @@ val generalize_expansive : Env.t -> type_expr -> unit contravariant branches non-generalizable *) val generalize_structure : type_expr -> unit -(* Same, but variables are only lowered to !current_level *) +(** Generalize eligible structural nodes, but lower their variables to + [current_level] instead of generalizing the variables. *) val correct_levels : type_expr -> type_expr (* Returns a copy with decreasing levels *) val instance : ?partial:bool -> Env.t -> type_expr -> type_expr +(** Instantiate a type scheme in a fresh copy session. + + Without [partial], generic nodes are copied and non-generic nodes are + shared. With [partial], non-generic subterms without free universal + variables are replaced by fresh variables: at [current_level] when false, + or at the original subterm's level when true. Structured subterms with free + universal variables are copied. Copies belonging to local GADT constraints + are recorded in [env], but that recording does not select additional nodes + for copying. *) -(* Take an instance of a type scheme *) -(* partial=None -> normal - partial=false -> newvar() for non generic subterms - partial=true -> newty2 ty.level Tvar for non generic subterms *) val instance_def : type_expr -> type_expr (* use defaults *) @@ -170,25 +176,28 @@ val instance_constructor : val instance_parameterized_type : ?keep_names:bool -> type_expr list -> type_expr -> type_expr list * type_expr val instance_declaration : type_declaration -> type_declaration + val instance_poly : ?keep_names:bool -> fixed:bool -> type_expr list -> type_expr -> type_expr list * type_expr -(* Instantiate a scheme [Tpoly(sch, univars)]: replace the universal - variables with fresh ones and return them with the instance. [~fixed] - controls the copy of polymorphic-variant rows: a fixed copy keeps their - rows closed to further extension. Scheme *use* sites instantiate with - [~fixed:false]; scheme *introduction* sites (checking a value against - the scheme) instantiate with [~fixed:true] and then verify the value - generalizes over the returned variables ([Typecore.check_univars]) - - the introduction discipline is that whole operation, not this flag. *) -(* Take an instance of a type scheme containing free univars *) +(** [instance_poly ~fixed univars body] replaces the universal variables of a + [Tpoly(body, univars)] scheme with fresh variables and returns those + variables with the copied body. + + [fixed] controls how fixed polymorphic-variant rows are copied; when true, + the copied rows remain closed to extension. It does not by itself select + scheme introduction or elimination. A type-checking caller introducing a + value at the scheme must separately verify generality. Callers in Typecore + do this with the local [check_univars] operation. *) val instance_label : bool -> label_description -> type_expr list * type_expr * type_expr -(* Same, for a label *) +(** Instantiate a record label's argument and result types. The Boolean is the + [fixed] argument passed to [instance_poly] when the argument is a + polymorphic scheme. *) val apply : Env.t -> type_expr list -> type_expr -> type_expr list -> type_expr (* [apply [p1...pN] t [a1...aN]] match the arguments [ai] to @@ -233,12 +242,24 @@ val filter_arrow_n : parameters with the given labels; return parameter and result types. *) val filter_method : Env.t -> string -> type_expr -> type_expr +(** Constrain a type to have the named readable object field and return the + field's stored type. If lookup reaches a [Tvar] row terminator, a missing + field extends that inferred row as [Immutable]. A row ending in [Tunivar], + [Tconstr], or [Tnil] cannot acquire a missing field. A non-object type which + cannot be constrained to an object also raises [Unify]. *) type object_field_write_error = Owrite_missing | Owrite_not_mutable val filter_object_field_for_write : Env.t -> string -> type_expr -> (type_expr, object_field_write_error) Result.t -(* A special case of unification (with {m : 'a; 'b}). *) +(** Constrain a type for assignment to the named object field. + + A mutable field returns its stored type. When the object row ends in a + [Tvar], an immutable field is promoted without changing its type, and a + missing field is added as mutable. [Tunivar] and private-row [Tconstr] + terminators are structurally open but cannot be strengthened. With either + of those terminators, or with [Tnil], an immutable field returns + [Owrite_not_mutable] and a missing field returns [Owrite_missing]. *) val occur_in : Env.t -> type_expr -> type_expr -> bool val deep_occur : type_expr -> type_expr -> bool @@ -261,13 +282,17 @@ val equal : Env.t -> bool -> type_expr list -> type_expr list -> bool [/\x1.../\xn.tau] and [/\y1.../\yn.sigma] are equivalent. *) val enlarge_type : Env.t -> type_expr -> type_expr * bool -(* Make a type larger, flag is true if some pruning had to be done *) +(** Build a supertype approximation used by the first coercion-checking path. + The result may share unchanged nodes with its input. Changed object fields + have independent mutability cells so trial unification cannot change the + input. The flag reports that a more complex coercion may do better. *) val subtype : Env.t -> type_expr -> type_expr -> unit -> unit -(* [subtype env t1 t2] checks that [t1] is a subtype of [t2]. - It accumulates the constraints the type variables must - enforce and returns a function that enforces this - constraints. *) +(** [subtype env t1 t2] checks that [t1] is a subtype of [t2] and returns a + function which enforces the type-variable equality constraints accumulated + during traversal. Callers can bind variables to their actual types before + invoking the returned function. Specialized cases may perform speculative + unification while constructing the constraints. *) val nondep_type : Env.t -> Ident.t -> type_expr -> type_expr (* Return a type equivalent to the given type but without diff --git a/compiler/ml/subst.ml b/compiler/ml/subst.ml index b98a5a4a246..e10b278a26a 100644 --- a/compiler/ml/subst.ml +++ b/compiler/ml/subst.ml @@ -220,6 +220,9 @@ let rec typexp_rec s ty = } | None -> Tvariant row)) | _ -> + (* A graph prepared for persistence must not retain mutable cells + from the live typing graph. Fresh cells still use the enclosing + copy session, so sharing within the saved graph is preserved. *) copy_type_desc ~fresh_mutability:s.for_saving (typexp_rec s) desc); ty' diff --git a/compiler/ml/subst.mli b/compiler/ml/subst.mli index 62ed5d51ab6..3ab73865218 100644 --- a/compiler/ml/subst.mli +++ b/compiler/ml/subst.mli @@ -40,7 +40,13 @@ val add_type_function : val add_module : Ident.t -> Path.t -> t -> t val add_module_path : Path.t -> Path.t -> t -> t val add_modtype : Ident.t -> module_type -> t -> t + val for_saving : t -> t +(** Return a substitution mode for constructing persistent compiler data. + Types copied in this mode are independent from the live source graph: + object-field mutability classes are fresh, while sharing between fields in + the copied result is preserved. *) + val reset_for_saving : unit -> unit val module_path : t -> Path.t -> Path.t diff --git a/compiler/ml/typecore.ml b/compiler/ml/typecore.ml index d6f9ab1c60a..2e804ab5d38 100644 --- a/compiler/ml/typecore.ml +++ b/compiler/ml/typecore.ml @@ -2336,6 +2336,13 @@ let should_unify_expected_result_before_typing_lowered_apply funct sargs = | _ -> false type targs = (Asttypes.arg_label * Typedtree.expression option) list + +(* Eliminate an object-field scheme for a read. A declared field payload is + wrapped in [Tpoly]: an empty binder list needs ordinary instantiation, while + a nonempty list needs polymorphic instantiation. An inferred field whose + scheme is not known yet can still be a variable; constrain it to the + ordinary, empty-binder form. Assignment uses [type_object_field_value] to + introduce rather than eliminate a scheme. *) let object_field_use_type env typ = match Ctype.repr typ with | {desc = Tpoly (ty, [])} -> instance env ty @@ -3289,10 +3296,15 @@ and type_expect_ ?deprecated_context ~context ?(recarg = Rejected) env sexp gen) else true in + (* Coercion has two checking paths. For a suitable non-generalizable + source, first try unification with an enlarged approximation of the + target under a snapshot. Otherwise, or if that trial fails, generate + and enforce subtype constraints. A type constructor whose variance or + capabilities affect coercion must therefore be represented correctly + both by [Ctype.enlarge_type] and by [Ctype.subtype]. *) (if (not gen) && - (* first try a single coercion *) let snap = snapshot () in let ty, _b = enlarge_type env ty' in try @@ -3368,6 +3380,8 @@ and type_expect_ ?deprecated_context ~context ?(recarg = Rejected) env sexp | Pexp_object_get (e, ({txt = met} as met_loc)) -> ( let obj = type_exp ~context:None env e in try + (* Lookup constrains an open object row to contain the field, without + requiring or adding write capability. *) let typ = filter_method env met obj.exp_type in let typ = object_field_use_type env typ in rue @@ -3388,6 +3402,9 @@ and type_expect_ ?deprecated_context ~context ?(recarg = Rejected) env sexp (obj.exp_type, met, object_valid_fields env obj.exp_type) ))) | Pexp_object_set (e, ({txt = name} as name_loc), svalue) -> ( let obj = type_exp ~context:None env e in + (* Assignment can add or promote a field only while its object row is + open. The result distinguishes a missing field from a closed immutable + field so the two cases retain their respective diagnostics. *) let field_typ = try Ctype.filter_object_field_for_write env name obj.exp_type with Unify _ -> Error Ctype.Owrite_missing @@ -3980,18 +3997,15 @@ and type_label_access env srecord lid = in (record, label, opath) -(* Typing the right-hand side of an object-field assignment: the - introduction dual of [object_field_use_type]. A field's type is a scheme: - reading instantiates it, while writing must establish it, so a - polymorphic field only accepts a value at least as polymorphic — checked - by typing the value at a fixed instance and verifying it generalizes - ([instance_poly true] + [check_univars]), the same discipline as - [type_label_exp] for record labels and [type_let] for polymorphic - annotations. With no quantified variables, establishing and instantiating - the scheme coincide. [type_label_exp] additionally retries an expansive - value without type propagation (PR#4862); that is a label-specific - completeness recovery, not part of the scheme-introduction contract, and - is deliberately not replicated here. *) +(* Introduce the right-hand side of an object-field assignment at the field's + scheme. A polymorphic field accepts only a value at least as general as the + scheme: type the value at a fixed instance, verify generality with + [check_univars], then replace the expression type with an ordinary instance. + [type_label_exp] for record labels and [type_let] for polymorphic annotations + use the same discipline. With no quantified variables, introduction and + elimination coincide. [type_label_exp] also retries an expansive value + without type propagation (PR#4862); that label-specific completeness + recovery is not part of scheme introduction and is not used here. *) and type_object_field_value env svalue typ = match (Ctype.repr typ).desc with | Tpoly (ty, (_ :: _ as tl)) -> diff --git a/compiler/ml/types.mli b/compiler/ml/types.mli index 0105d01f479..a7f62419f23 100644 --- a/compiler/ml/types.mli +++ b/compiler/ml/types.mli @@ -53,19 +53,6 @@ type type_expr = {mutable desc: type_desc; mutable level: int; id: int} all other known types. Whereas [type_expr] is a pure construct which allows referring to existing types. - - Note on object-field mutability: each [Tfield] carries a - [field_mutability ref]. Semantically the state is two-valued - ([Immutable] or [Mutable], read through [Btype.mutability_repr]); - [Mutability_link] is pure representation — a union-find edge that makes - every field constrained to have the same mutability share one - equivalence class, so a promotion (Immutable to Mutable on an open row) - is seen by all of them at once. Classes are merged only by unification; - promotions and merges are logged on the backtracking trail; copying - duplicates a class iff the row ends in a generic variable (a scheme - instantiation) and shares it otherwise, mirroring the sharing law of - the row variable itself. Links never appear in saved (marshalled) - types. *) and arg = {lbl: arg_label; typ: type_expr} @@ -94,8 +81,8 @@ and type_desc = rest: type_expr; } (** [Tfield {name = "foo"; mutability; typ; rest}] - ==> [{.. "foo": typ, rest}]; [mutability] records whether the field - admits assignment ([@set]). *) + ==> [{.. "foo": typ, rest}]. [mutability] is a shared cell; inspect its + semantic value with [Btype.mutability_repr]. *) | Tnil (** [Tnil] ==> [<...; >] *) | Tlink of type_expr (** Indirection used by unification engine. *) | Tsubst of type_expr (* for copying *) @@ -182,10 +169,12 @@ and abbrev_memo = | Mlink of abbrev_memo ref (** Abbreviations can be found after this indirection *) -(** Mutability state of an object field, shared through an equivalence class - of cells. [Mutability_link] is an internal graph edge (union by - unification, duplication memo during copying) — never a third semantic - state; the class's value is at the end of the link chain. *) +(** Assignment capability of an object field. + + The semantic value is [Immutable] or [Mutable]. [Mutability_link] is an + internal edge between cells which have the same capability, not a third + value. Code making a semantic decision must use [Btype.mutability_repr] + rather than inspect a cell directly. *) and field_mutability = | Mutability_value of Asttypes.mutable_flag | Mutability_link of field_mutability ref diff --git a/compiler/syntax/Formatter.md b/compiler/syntax/Formatter.md new file mode 100644 index 00000000000..79e264368b6 --- /dev/null +++ b/compiler/syntax/Formatter.md @@ -0,0 +1,25 @@ +# Formatter policy + +The ReScript formatter is deliberately opinionated and has no formatting +configuration. The core team chooses formatting behavior to keep ReScript code +consistent across projects and to keep the printer maintainable. + +Bug reports and proposals are welcome, but formatter changes are evaluated by +technical correctness, consistency with the language, implementation +complexity, and core-team consensus. Similar behavior in another formatter is +useful evidence, not by itself a reason to adopt a rule. + +Some constructs preserve a source author's choice to make them multiline. For +example, the printer preserves meaningful line breaks in pipe chains and +records. These decisions describe current behavior rather than a general rule +that all source line breaks must be retained. + +When changing the formatter: + +- test both narrow and wide print widths; +- cover comments and the parentheses needed to preserve parsing; +- run `make test-syntax` and `make test-syntax-roundtrip`; +- inspect snapshot changes for unrelated reformatting; +- prefer a simple rule that behaves consistently across equivalent AST shapes. + +The implementation map and diagnostic commands are in [README.md](README.md). diff --git a/compiler/syntax/JSX.md b/compiler/syntax/JSX.md new file mode 100644 index 00000000000..fc6f5ecf27a --- /dev/null +++ b/compiler/syntax/JSX.md @@ -0,0 +1,67 @@ +# Built-in JSX transformation + +This document describes the current compiler-facing JSX transformation. For +language usage and project configuration, use the +[JSX configuration manual](https://rescript-lang.org/docs/manual/build-configuration/#jsx) +and the [ReScript React documentation](https://rescript-lang.org/docs/react/beyond-jsx/). + +## Entry points and configuration + +[`jsx_ppx.ml`](src/jsx_ppx.ml) applies the built-in transformation to +implementations and signatures. It receives the project JSX version and module +from the compiler driver. A `@jsxConfig` structure or signature attribute can +override `version` and `module_` for the following items in that scope; nested +structures and signatures save and restore the enclosing configuration. + +The current transformation is version 4, implemented by +[`jsx_v4.ml`](src/jsx_v4.ml). A non-React JSX module selects the same transform +and changes the module paths emitted by it. Configuration state is local to a +mapper invocation and includes the current nested-module path and whether the +scope already defines a component. + +## Component definitions + +`@react.component` and `@jsx.component` mark a component definition. The +transform validates its labelled parameters and generates the props type and +wrapper required by the JSX runtime. Only one component definition is allowed +in a module; additional components must be placed in nested modules or +separate files. + +The same transformation is applied to signatures so the implementation and +interface expose compatible component types. External components and the +legacy `componentWithProps` form have separate validation paths; tests for +these forms live under `tests/build_tests/react_ppx/`. + +## JSX expressions + +The parser represents JSX explicitly in the parsetree. The transformation +rewrites those nodes as calls to the configured JSX module: + +- an uppercase tag normally denotes the module's `make` component; +- a qualified tag whose final component is lowercase denotes that value + directly, which supports external components; +- a lowercase tag is emitted through the configured host-element module; +- fragments use the configured `jsxFragment` value; +- one child becomes a `children` prop and multiple children become an array; +- keyed elements select the keyed runtime entry point; +- at most one props spread is accepted, and it must precede explicit props. + +The precise runtime entry points differ between React host elements and a +generic JSX module. Read `mk_react_jsx`, `append_children_prop`, and +`mk_uppercase_tag_name_expr` in `jsx_v4.ml` together when changing expression +lowering. + +## Change checklist + +A change to this transform normally needs all of the following: + +- implementation and signature handling kept in agreement; +- React and generic-module cases checked separately; +- parser/printing coverage if the surface JSX shape changes; +- transform snapshots under `tests/syntax_tests/`; +- component type and diagnostic coverage under `tests/build_tests/`; +- analysis and GenType coverage when the generated component shape changes. + +Use `dune exec res_parser -- -jsx-version 4 -print ml example.res` to inspect +the transformed parsetree during development. The CLI flag is a diagnostic +interface, not a supported project configuration mechanism. diff --git a/compiler/syntax/README.md b/compiler/syntax/README.md new file mode 100644 index 00000000000..6c2aeb9e2fd --- /dev/null +++ b/compiler/syntax/README.md @@ -0,0 +1,79 @@ +# Parser, printer, and JSX transformation + +This directory owns ReScript source parsing, comment attachment, printing, and +the built-in JSX transformation. The parser is hand-written and produces the +parsetree consumed by `compiler/ml`. + +## Code map + +- [`src/res_scanner.ml`](src/res_scanner.ml) tokenizes source text. +- [`src/res_parser.ml`](src/res_parser.ml) and + [`src/res_grammar.ml`](src/res_grammar.ml) implement parsing and recovery. +- [`src/res_comment.ml`](src/res_comment.ml) and + [`src/res_comments_table.ml`](src/res_comments_table.ml) retain and attach + comments for printing. +- [`src/res_printer.ml`](src/res_printer.ml), + [`src/res_doc.ml`](src/res_doc.ml), and + [`src/res_parens.ml`](src/res_parens.ml) implement formatting. +- [`src/jsx_ppx.ml`](src/jsx_ppx.ml) selects and applies the built-in JSX + transformation; [`src/jsx_v4.ml`](src/jsx_v4.ml) implements the current + transform. +- [`cli/res_cli.ml`](cli/res_cli.ml) provides the repository-only `res_parser` + diagnostic tool. Production compiler code calls the syntax library APIs. + +See [Formatter.md](Formatter.md) for formatter policy and [JSX.md](JSX.md) for +the current JSX transformation contract. + +## Building and testing + +Run commands from the repository root: + +```sh +make # build the compiler and build system +make test-syntax # parser and printer tests +make test-syntax-roundtrip # parse/print round-trip tests +make checkformat # check repository formatting +``` + +Use the repository diagnostic CLI to inspect one file: + +```sh +dune exec res_parser -- example.res +dune exec res_parser -- -print tokens example.res +dune exec res_parser -- -print ast -recover example.res +dune exec res_parser -- -print comments example.res +dune exec res_parser -- -print ml example.res +dune exec res_parser -- -print res -width 80 example.res +``` + +The CLI is for compiler development and tests; it is not a supported public +parser interface. + +## Changing syntax + +A syntax change can affect more than the grammar. Check each relevant layer: + +1. scanner tokens and parser recovery; +2. the current parsetree in `compiler/ml/parsetree.ml`; +3. printing, parentheses, and comment attachment; +4. the v0 AST bridges in `compiler/ml/ast_mapper_from0.ml` and + `compiler/ml/ast_mapper_to0.ml`; +5. type checking and every later compiler representation that carries the + construct; +6. parser, round-trip, type-error, and end-to-end tests. + +Do not modify `compiler/ml/parsetree0.ml`. It is the frozen input/output shape +for existing PPX integrations. When the current parsetree changes, define an +explicit compatibility mapping in both directions; do not use a wildcard to +discard a new construct. + +Parser tests should cover valid input, recovery from invalid input, printing, +and comment placement where applicable. Run the round-trip suite whenever a +change affects parsing or printing, even when the intended AST is unchanged. + +## Documentation placement + +Public language behavior belongs on the ReScript website. This directory keeps +implementation-facing documentation needed to change the parser, printer, or +built-in transformations. Put API contracts in `.mli` files and local parsing +or printing invariants beside their implementation. diff --git a/context7.json b/context7.json index 64f80d26e54..07aed6bc754 100644 --- a/context7.json +++ b/context7.json @@ -37,7 +37,6 @@ "migrate.ml", "migrate.mli", "package.json", - "reactive_reanalyze_design.md", "rescript.json", "tools.ml", "transforms.ml", diff --git a/docs/Formatter.md b/docs/Formatter.md deleted file mode 100644 index a157b52a422..00000000000 --- a/docs/Formatter.md +++ /dev/null @@ -1,48 +0,0 @@ -# ReScript Formatter - -## Philosophy - -The ReScript formatter is **opinionated**. Formatting decisions are made by the core team based on our collective judgment and vision for the language. We do not aim to accommodate every stylistic preference or engage in extended debates about formatting choices. - -The formatter currently has **no configuration settings**, and we aspire to keep it that way. This ensures that ReScript code looks consistent across all projects and teams, eliminating style debates and configuration overhead. - -## Decision Making - -- **Core team consensus is final**: When the core team reaches consensus on a formatting decision, that decision stands. There is no requirement for community-wide agreement or extensive discussion. - -- **Community input is welcome but not binding**: We appreciate suggestions and feedback from the community, but these can be closed without extensive justification if the core team is not aligned with the proposal. - -- **No endless style discussions**: We are not interested in protracted debates about formatting preferences. The formatter exists to provide consistent, automated formatting—not to serve as a platform for style negotiations. - -## Prior Decisions - -The following are examples of formatting decisions the core team has made. This list is not exhaustive, and these decisions do not create binding precedents for future discussions. The core team retains full discretion to make different decisions in similar cases. - -- **Smart linebreaks for pipe chains**: The formatter preserves user-introduced linebreaks in pipe chains (`->`), allowing users to control multiline formatting. See [forum announcement](https://forum.rescript-lang.org/t/ann-smart-linebreaks-for-pipe-chains/4734). - -- **Preserve multilineness for records**: The formatter preserves multiline formatting for record types and values when users introduce linebreaks. See [issue #7961](https://github.com/rescript-lang/rescript/issues/7961). - -**Important**: These examples are provided for reference only. They do not establish rules or precedents that constrain future formatting decisions. The core team may choose different approaches in similar situations based on current consensus. - -## Guidelines for Contributors - -### Submitting Formatting Issues - -- You may open issues to report bugs or propose improvements -- Understand that proposals may be closed if they don't align with core team vision -- Avoid reopening closed issues unless there's new technical information -- Respect that "the core team isn't feeling it" is a valid reason for closure - -### What We Consider - -- Technical correctness and consistency -- Alignment with ReScript's design philosophy -- Maintainability and simplicity of the formatter implementation -- Core team consensus - -### What We Generally Avoid - -- Style preferences that don't align with our vision -- Using comparisons to other formatters as the sole justification for changes (while we may align with other formatters on many decisions, we make choices based on our own judgment, not because another formatter does it) -- Requests that would significantly complicate the formatter implementation -- Debates about subjective formatting choices diff --git a/docs/JSXV4.md b/docs/JSXV4.md index d7657e2c88b..2f90b9f64d8 100644 --- a/docs/JSXV4.md +++ b/docs/JSXV4.md @@ -1,435 +1,13 @@ -## Introduction - -JSX V4, supported in the compiler version introduces a new idiomatic record-based representation of components which is incompatible with V3. Because of this, either the entire project or dependencies need to be compiled in V4 mode, or some compatibility features need to be used to mix V3 and V4 in the same project. -The V4 representation is part of the spec, so `@react.component` is effectively just an abbreviation for code that can be written by hand. - -## Turn On V4 - -To build an entire project in V4 mode, including all its dependencies, use the new `"jsx"` configuration in `rescript.json` instead of the old `"reason"`: - -```json -"jsx": { "version": 4 } -``` - -> Note that JSX V4 requires the rescript compiler 10.1 or higher, and `rescript-react` version `0.11` or higher. In addition, `react` version `18.0` is required. - -## Configuration And Upgrade - -### Dependency-level config - -Dependencies inherit the `jsx` configuration of the root project. So if the root project uses V4 then the dependencies are built using V4, and the same for V3. -To build certain dependencies in V3 compatibility mode, whatever the version used in the root project, use `"v3-dependencies"`: the listed dependencies will be built in V3 mode, and in addition `-open ReactV3` is added to the compiler options. - -For example, suppose a V3 project uses rescript-react 0.11, which requires compatibility mode if compiled with V3, and that 2 dependencies `"rescript-react-native", "rescript-react-navigation"` only build with compatibility mode. Then the setting will be: - -```json -"jsx": { - "version": 3, - "v3-dependencies": ["rescript-react-native", "rescript-react-navigation"] -}, -"compiler-flags": ["-open ReactV3"] -``` - -Another example is a V4 project that also uses `"rescript-react-native", "rescript-react-navigation"`. Then the setting will be: - -```json -"jsx": { - "version": 4, - "v3-dependencies": ["rescript-react-native", "rescript-react-navigation"] -} -``` - -> Note: do not add @rescript/react to the v3-dependencies, or it will cause a cyclic dependencies error. - -### Classic and Automatic Mode - -Classic mode generates calls to `React.createElement` just as with V3. - -```json -"jsx": { - "version": 4, - "mode": "classic" -} -``` - -Automatic mode is the default and generates calls to `_jsx` functions (similar to TypeScript's `react-jsx` mode) - -```json -"jsx": { - "version": 4, - "mode": "automatic" -} -``` - -### File-level config - -The top-level attribute `@@jsxConfig` is used to update the `jsx` config for the rest of the file (or until the next config update). Only the values mentioned are updated, the others are left unchanged. - -```rescript -@@jsxConfig({ version: 4, mode: "automatic" }) - -module Wrapper = { - module R1 = { - @react.component // V4 and new _jsx transform - let make = () => body - } - - @@jsxConfig({ version: 4, mode: "classic" }) - - module R2 = { - @react.component // V4 with `React.createElement` - let make = () => body - } -} - -@@jsxConfig({ version: 3 }) - -@react.component // V3 -let make = () => body -``` - -### Migration of V3 components that depend on the internal representation - -Some components in existing projects are written in a way that is dependent on the V3 internal representation. -Here are a few examples of how to convert them to V4. - -#### `makeProps` does not exist in V4 - -Rewrite this: - -```rescript -// V3 -module M = { - @obj external makeProps: (~msg: 'msg, ~key: string=?, unit) => {"msg": 'msg} = "" // No more makeProps - - let make = (~msg) => { -
{React.string(msg)}
- } -} -``` - -To this: - -```rescript -// V4 -module M = { - type props<'msg> = {msg: 'msg} - let make = props =>
{React.string(props.msg)}
-} -``` - -#### `React.Context` - -Rewrite this: - -```rescript -module Context = { - let context = React.createContext(() => ()) - - module Provider = { - let provider = React.Context.provider(context) - - @react.component - let make = (~value, ~children) => { - React.createElement(provider, {"value": value, "children": children}) // Error - } - } -} -``` - -To this: - -```rescript -module Context = { - let context = React.createContext(() => ()) - - module Provider = { - let make = React.Context.provider(context) - } -} -``` - -#### `React.forwardRef`(Discouraged) - -`forwardRef` is discouraged, but sometimes used in existing V3 code such as this example: - -```rescript -module FancyInput = { - @react.component - let make = React.forwardRef(( - ~className=?, - ~children, - ref_, // argument - ) => -
- Nullable.toOption->Option.map(ReactDOM.Ref.domRef)} - /> - children -
- ) -} - -@react.component -let make = () => { - let input = React.useRef(Nullable.null) - -
- // prop - - -
-} -``` - -In this example, there is an inconsistency between `ref` as prop and `ref_` as argument. With JSX V4, `ref` is only allowed as an argument. - -```rescript -module FancyInput = { - @react.component - let make = React.forwardRef(( - ~className=?, - ~children, - ref, // only `ref` is allowed - ) => -
- Nullable.toOption->Option.map(ReactDOM.Ref.domRef)} - /> - children -
- ) -} - -@react.component -let make = () => { - let input = React.useRef(Nullable.null) - -
- - - -
-} -``` - -## V4 Spec - -This is the specification that decribes the two JSX V4 transformations: - -- For component definition `@react.component let make = ...` -- For component application `` - -The transformations are optional in that it is possible to write the resulting code manually instead of using them. - -### Pre-transformation for component definition - -To simplify the description of component definition, a pre-transformation -is used to move `@react.component` to a place where the actual transformations operate. - -#### Normal Case - -```rescript -@react.component -let make = (~x, ~y, ~z) => body -``` - -is pre-transformed to - -```rescript -let make = @react.component (~x, ~y, ~z) => body -``` - -#### Forward Ref - -```rescript -@react.component -let make = React.forwardRef((~x, ~y, ref) => body) -``` - -is pre-transformed to - -```rescript -let make = React.forwardRef({ - let fn = - @react.component (~x, ~y) => ref => body - (props, ref) => fn(props, ref) -}) -``` - -### Transformation for Component Definition - -```rescript -@react.component (~x, ~y=3+x, ~z=?) => body -``` - -is transformed to - -```rescript -type props<'x, 'y, 'z> = {x: 'x, y?: 'y, z?: 'z} - -({x, ?y, ?z}: props<_, _, _>): Jsx.element => { - let y = switch y { - | None => 3 + x - | Some(y) => y - } - body -} -``` - -> Note: -> - This implicit definition of type `props` means that there cannot be other type definitions of `props` in the same scope, or it will be a compiler error about multiple definitions of the type name. -> - JSX V4 automatically adds a `Jsx.element` return constraint to all component functions. This ensures that components always return valid JSX elements and provides better type safety. If a component returns a non-JSX value, you'll get a helpful error message suggesting how to convert it (e.g., use `React.int` for integers, `React.string` for strings, etc.). - -### Transformation for Component Application - -```rescript - -// is transformed to -React.createElement(Comp.make, {x: x}) - - -// is transformed to -React.createElement(Comp.make, {x, y: 7, ?z}) - - -// is transformed to -React.createElement(Comp.make, React.addKeyProp(~key="7", {x: x})) - - -// is transformed to -React.createElement(Comp.make, React.addKeyProp(~key=?Some("7"), {x: x})) -``` - -### New experimental automatic mode - -The V4 ppx supports [the new jsx transform](https://reactjs.org/blog/2020/09/22/introducing-the-new-jsx-transform.html) of React.js. - -The jsx transform only affects component application, but not the definition. - -```rescript - -// is transformed to -React.jsx(Comp.make, {x: x}) -``` - -```rescript -
-// is transformed to -ReactDOM.jsx("div", { name: "div" }) -``` - -The props type of dom elements, e.g. `div`, is inferred to `ReactDOM.domProps`. - -```rescript -type domProps = { - key?: string, - id?: string, - ... -} -``` - -### Interface And External - -```rescript -@react.component (~x: int, ~y: int=?, ~z: int=?) => React.element - -// is transformed to - -type props<'x, 'y, 'z> = {x: 'x, y?: 'y, z?: 'z} - -props => React.element -``` - -Since an external is a function declaration, it follows the same rule. - -### Component Name - -The convention for names is the same one used in V3: the generated -function has the name of the enclosing module/file. - -### Fragments - -```rescript -<> comp1 comp2 comp3 - -// is transformed to - -// v4 -React.createElement(React.fragment, {children: [comp1, comp2, comp3]}) - -// v4 @ new jsx transform -React.jsxs(React.jsxFragment, {children: [comp1, comp2, comp3]}) -``` - -### Spread props (new feature) - -V4 introduces support for the spread operator for props: `{...p}`. - -```rescript -module A = { - @react.component - let make = (~x, ~y) => body -} - -let p: A.props<_> = {x: "x", y: "y"} - - - - -// not allowed - - -``` - -### Shared props type (new feature) - -V4 introduces support to control the definition of the `props` type by passing as argument to `@react.component` the body of the type definition of `props`. The main application is sharing a single type definition across several components. Here are a few examples: - -```rescript -type sharedprops<'x, 'y> = {x: 'x, y: 'y, z:string} - -module C1 = { - @react.component(:sharedProps<'a, 'b>) - let make = (~x, ~y) => React.string(x ++ y ++ z) -} - -module C2 = { - @react.component(:sharedProps) - let make = (~x, ~y) => React.string(x ++ y ++ z) -} - -module C3 = { - type myProps = sharedProps - @react.component(:myProps) - let make = (~x, ~y) => React.int(x + y) -} -``` - -The generated code (some details removed) looks like this: - -```rescript -@@jsxConfig({version: 4, mode: "classic"}) - -type sharedprops<'x, 'y> = {x: 'x, y: 'y, z: string} - -module C1 = { - type props<'a, 'b> = sharedProps<'a, 'b> - let make = ({x, y, _}: props<_>) => React.string(x ++ y ++ z) -} - -module C2 = { - type props<'b> = sharedProps - let make = ({x, y, _}: props<_>) => React.string(x ++ y ++ z) -} - -module C3 = { - type myProps = sharedProps - type props = myProps - let make = ({x, y, _}: props) => React.int(x + y) -} -``` +# JSX version 4 + +This path is retained because older announcements and discussions link to it. +The previous document mixed the current transform with obsolete migration and +configuration material. + +- For current language usage and configuration, see the + [JSX configuration manual](https://rescript-lang.org/docs/manual/build-configuration/#jsx) + and the [ReScript React documentation](https://rescript-lang.org/docs/react/beyond-jsx/). +- For migration from older ReScript releases, see the + [version 12 migration guide](https://rescript-lang.org/docs/manual/migrate-to-v12/). +- For compiler implementation details, see + [`compiler/syntax/JSX.md`](../compiler/syntax/JSX.md). diff --git a/docs/Syntax.md b/docs/Syntax.md deleted file mode 100644 index 6486fc2acda..00000000000 --- a/docs/Syntax.md +++ /dev/null @@ -1,124 +0,0 @@ -# ReScript Syntax - -Documentation: https://rescript-lang.org/docs/manual/latest/overview - -## Contribute - -### Why - -A detailed discussion by Jonathan Blow and Casey Muratori on why you would hand-roll a parser for a production quality programming language -[Discussion: Making Programming Language Parsers, etc](https://youtu.be/MnctEW1oL-E) - -"One reason why I switched off these parser tools is that the promises didn't really materialize. -The amount of work that I had to do change a yacc script from one language to a variant of that language -was more than if I hand wrote the code myself. -" -J. Blow. - -### Setup & Usage - -Required: - -- OCaml 4.10 or later -- OS: macOS, Linux or Windows - -In the root of the rescript-compiler repo, run - -```sh -opam install . --deps-only --with-test -``` - -To build the syntax sources, run - -```sh -make # or "dune build" -``` - -This will produce the three binaries `res_parser`, `syntax_tests` and `syntax_benchmarks` (with `.exe` extension on Windows). - -We only build production binaries, even in dev mode. No need for a separate dev binary when the build is fast enough. Plus, this encourages proper benchmarking of the (production) binary each diff. - -After you make a change: - -```sh -make -``` - -Run the core tests: - -```sh -make test -``` - -Run the extended tests (not fully working on Windows yet): - -```sh -make roundtrip-test -``` - -Those will tell you whether you've got a test output difference. If it's intentional, check them in. - -Debug a file: - -```sh -# write code in test.res -dune exec -- res_parser test.res # test printer -dune exec -- res_parser -print ast test.res # print ast -dune exec -- res_parser -print comments test.res # print comment table -dune exec -- res_parser -print ml test.res # show ocaml code -dune exec -- res_parser -print res -width 80 test.res # test printer and change default print width -``` - -Benchmark: - -```sh -make bench -``` - -Enable stack trace: - -```sh -# Before you run the binary -export OCAMLRUNPARAM="b" -``` - -This is likely a known knowledge: add the above line into your shell rc file so that every shell startup you have OCaml stack trace enabled. - -### Development Docs - -#### Folder Structure - -- `src` contains all the parser/printer source code. Don't change folder structure without notice. -- `benchmarks`, `cli` and `tests` contain the source code for the executables used for testing/benchmarking. - -#### Error Reporting Logic - -Right now, ReScript's compiler's error reporting mechanism, for architectural reasons, is independent from this syntax repo's error reporting mechanism. However, we do want a unified look when they report the errors in the terminal. This is currently achieved by (carefully...) duplicating the error report logic from the compiler repo to here (or vice-versa; either way, just keep them in sync). The files to sync are the compiler repo's [super_location.ml](https://github.com/rescript-lang/rescript-compiler/blob/fcb21790dfb0592f609818df7790192061360631/jscomp/super_errors/super_location.ml) and [super_code_frame.ml](https://github.com/rescript-lang/rescript-compiler/blob/fcb21790dfb0592f609818df7790192061360631/jscomp/super_errors/super_code_frame.ml), into this repo's [res_diagnostics_printing_utils.ml](https://github.com/rescript-lang/syntax/blob/ec5cefb23b659b0a7be170ae0ad26f3fe8a05456/src/res_diagnostics_printing_utils.ml). A few notes: - -- Some lines are lightly changed to fit this repo's needs; they're documented in the latter file. -- Please keep these files lightweight and as dependency-less as possible, for easier syncing. -- The syntax logic currently doesn't have warnings, only errors, and potentially more than one. -- In the future, ideally, the error reporting logic would also be unified with GenType and Reanalyze's. It'd be painful to copy paste around all this reporting logic. -- The errors are reported by the parser [here](https://github.com/rescript-lang/syntax/blob/ec5cefb23b659b0a7be170ae0ad26f3fe8a05456/src/res_diagnostics.ml#L146). -- Our editor plugin parses the error report from the compiler and from the syntax [here](https://github.com/rescript-lang/rescript-vscode/blob/0dbf2eb9cdb0bd6d95be1aee88b73830feecb5cc/server/src/utils.ts#L129-L329). - -### Example API usage - -```ocaml -let filename = "foo.res" -let src = FS.readFile filename - -let p = - (* intended for ocaml compiler *) - let mode = Res_parser.ParseForTypeChecker in - (* if you want to target the printer use: let mode = Res_parser.Default in*) - Res_parser.make ~mode src filename - -let structure = Res_core.parseImplementation p -let signature = Res_core.parseSpecification p - -let () = match p.diagnostics with -| [] -> () (* no problems *) -| diagnostics -> (* parser contains problems *) - Res_diagnostics.printReport diagnostics src -``` diff --git a/docs/TYPING.adoc b/docs/TYPING.adoc deleted file mode 100644 index 8633ef52ed6..00000000000 --- a/docs/TYPING.adoc +++ /dev/null @@ -1,58 +0,0 @@ -The implementation of the OCaml typechecker is complex. Modifying it -will need a good understanding of the OCaml type system and type -inference. Here is a reading list to ease your discovery of the -typechecker: - -http://caml.inria.fr/pub/docs/u3-ocaml/index.html[Using, Understanding, and Unraveling the OCaml Language by Didier Rémy] :: -This book provides (among other things) a formal description of parts -of the core OCaml language, starting with a simple Core ML. - -http://okmij.org/ftp/ML/generalization.html[Efficient and Insightful Generalization by Oleg Kiselyov] :: -This article describes the basis of the type inference algorithm used -by the OCaml type checker. It is a recommended read if you want to -understand the type-checker codebase, in particular its handling of -polymorphism/generalization. - -After that, it is best to dive right in. There is no real "entry -point", but an understanding of both the parsetree and the typedtree -is necessary. - -The datastructures :: -link:types.mli[Types] and link:typedtree.mli[Typedtree] -are the two main datastructures in the typechecker. They correspond to -the source code annotated with all the information needed for type -checking and type inference. link:env.mli[Env] contains all the -environments that are used in the typechecker. Each node in the -typedtree is annotated with the local environment in which it was -type-checked. - -Core utilities :: -link:btype.mli[Btype] and link:ctype.mli[Ctype] contain -the various low-level function needed for typing, in particular -related to levels, unification and -backtracking. link:mtype.mli[Mtype] contains utilities related -to modules. - -Inference and checking:: -The `Type..` modules are related to inference and typechecking, each -for a different part of the language: -link:typetexp.mli[Typetexp] for type expressions, -link:typecore.mli[Typecore] for the core language, -link:typemod.mli[Typemod] for modules, -link:typedecl.mli[Typedecl] for type declarations and finally -link:typeclass.mli[Typeclass] for the object system. - -Inclusion/Module subtyping:: -Handling of inclusion relations are separated in the `Include...` -modules: link:includecore.ml[Includecore] for the type and -value declarations, link:includemod.mli[Includemod] for modules -and finally link:includeclass.mli[Includeclass] for the object -system. - -Dependencies between modules:: -Most of the modules presented above are inter-dependent. Since OCaml -does not permit circular dependencies between files, the -implementation uses forward declarations, implemented with references -to functions that are filled later on. An example can be seen in -link:typecore.ml[Typecore.type_module], which is filled in -link:typemod.ml[Typemod]. diff --git a/docs/reactive_reanalyze_design.md b/docs/reactive_reanalyze_design.md deleted file mode 100644 index cf828df65f4..00000000000 --- a/docs/reactive_reanalyze_design.md +++ /dev/null @@ -1,452 +0,0 @@ -# Reactive Reanalyze: Incremental Analysis Design Notes - -## Executive Summary - -This document is an early design exploration of making `reanalyze` incremental: keeping state between runs and reacting to file changes. - -**Note**: The current implementation lives in `analysis/reactive/` and `analysis/reanalyze/src/Reactive*` and differs from some details below (e.g. no `-parallel` flag; caching is implemented in OCaml, not via a C++ `Marshal_cache`). - -## Current Architecture - -### Reanalyze Processing Flow - -``` - ┌─────────────────┐ - │ Collect CMT │ - │ File Paths │ - └────────┬────────┘ - │ - ┌────────▼────────┐ - │ Load CMT Files │ ← 77% of time (~780ms) - │ (Cmt_format. │ - │ read_cmt) │ - └────────┬────────┘ - │ - ┌────────▼────────┐ - │ Process Each │ - │ File → file_data│ - └────────┬────────┘ - │ - ┌─────────────────┴─────────────────┐ - │ │ - ┌────────▼────────┐ ┌────────▼────────┐ - │ Merge Builders │ │ Exception │ - │ (annotations, │ │ Results │ - │ decls, refs, │ └─────────────────┘ - │ cross_file, │ - │ file_deps) │ ← 8% of time (~80ms) - └────────┬────────┘ - │ - ┌────────▼────────┐ - │ Solve (DCE, │ ← 15% of time (~150ms) - │ optional args) │ - └────────┬────────┘ - │ - ┌────────▼────────┐ - │ Report Issues │ ← <1% of time - └─────────────────┘ -``` - -### Current Bottleneck - -From the benchmark (50 copies, ~4900 files): - -| Phase | Time | % of Total | -|-------|------|------------| -| File loading | ~779ms | ~77% | -| Merging | ~81ms | ~8% | -| Solving | ~146ms | ~15% | -| Total | ~1007ms | 100% | - -**CMT file loading is the dominant cost** because each file requires: -1. System call to open file -2. Reading marshalled data from disk -3. Unmarshalling into OCaml heap -4. AST traversal to extract analysis data - -## Proposed Architecture: Reactive Analysis Service - -### Design Goals - -1. **Persistent service** - Stay running and maintain state between analysis runs -2. **File watching** - React to file changes (create/modify/delete) -3. **Incremental updates** - Only process changed files -4. **Cached results** - Keep processed `file_data` in memory -5. **Fast iteration** - Sub-10ms response for typical edits - -### Integration via reactive collections - -The implementation uses reactive collections and file-backed collections to cache processed per-file results and propagate changes incrementally. - -#### `ReactiveFileCollection` - Delta-Based Processing - -```ocaml -(* Create collection that maps CMT paths to processed file_data *) -let cmt_collection = ReactiveFileCollection.create - ~process:(fun (cmt_infos : Cmt_format.cmt_infos) -> - (* This is called only when file changes *) - process_cmt_for_dce ~config cmt_infos - ) - -(* Initial load - process all files once *) -List.iter (ReactiveFileCollection.process_file cmt_collection) all_cmt_paths - -(* On file watcher event - only process changed files *) -(* In practice, reanalyze uses batch processing for bulk load and explicit remove/add - operations for churn testing. *) - -(* Get all processed data for analysis *) -let file_data_list = ReactiveFileCollection.values cmt_collection -``` - -### Service Architecture - -``` -┌────────────────────────────────────────────────────────────────┐ -│ Reanalyze Service │ -├────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌──────────────┐ ┌─────────────────────────────────┐ │ -│ │ File Watcher │─────▶│ Reactive_file_collection │ │ -│ │ (fswatch/ │ │ ┌───────────────────────────┐ │ │ -│ │ inotify) │ │ │ path → file_data cache │ │ │ -│ └──────────────┘ │ │ (backed by Marshal_cache) │ │ │ -│ │ └───────────────────────────┘ │ │ -│ └──────────┬──────────────────────┘ │ -│ │ │ -│ │ file_data_list │ -│ ▼ │ -│ ┌─────────────────────────────────┐ │ -│ │ Incremental Merge & Solve │ │ -│ │ (may be reactive in future) │ │ -│ └──────────┬──────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────┐ │ -│ │ Issues / Reports │ │ -│ └─────────────────────────────────┘ │ -│ │ -└────────────────────────────────────────────────────────────────┘ -``` - -### API Design - -```ocaml -module ReactiveReanalyze : sig - type t - (** A reactive analysis service *) - - val create : config:DceConfig.t -> project_root:string -> t - (** Create a new reactive analysis service *) - - val start : t -> unit - (** Start file watching and initial analysis *) - - val stop : t -> unit - (** Stop file watching *) - - val analyze : t -> AnalysisResult.t - (** Run analysis on current state. Fast if no files changed. *) - - val on_file_change : t -> string -> unit - (** Notify of a file change (for external file watchers) *) - - val apply_events : t -> Reactive_file_collection.event list -> unit - (** Apply batch of file events *) -end -``` - -## Performance Analysis - -### Expected Speedup - -| Scenario | Current | With skip-lite | Speedup | -|----------|---------|----------------|---------| -| Cold start (all files) | 780ms | 780ms | 1x | -| Warm cache, no changes | 780ms | ~20ms | **39x** | -| Single file changed | 780ms | ~2ms | **390x** | -| 10 files changed | 780ms | ~15ms | **52x** | - -### How skip-lite Achieves This - -1. **Marshal_cache.with_unmarshalled_if_changed**: - - Stats all files to check modification time (~20ms for 5000 files) - - Only unmarshals files that changed - - Returns `None` for unchanged files, `Some result` for changed - -2. **Reactive_file_collection**: - - Maintains hash table of processed values - - On `apply`, only processes files in the event list - - Iteration is O(n) but values are already computed - -### Memory Considerations - -| Data | Storage | GC Impact | -|------|---------|-----------| -| CMT file bytes | mmap (off-heap) | None | -| Unmarshalled cmt_infos | OCaml heap (temporary) | During callback only | -| Processed file_data | OCaml heap (cached) | Scanned by GC | - -For 5000 files with average 20KB each: -- mmap cache: ~100MB (off-heap, OS-managed) -- file_data cache: ~50MB (on-heap, estimate) - -## Implementation Plan - -### Phase 1: Integration Setup - -1. **Add skip-lite dependency** to dune/opam -2. **Create wrapper module** `CmtCache` that provides: - ```ocaml - val read_cmt : string -> Cmt_format.cmt_infos - (** Drop-in replacement for Cmt_format.read_cmt using Marshal_cache *) - ``` - -### Phase 2: Reactive Collection - -1. **Define file_data type** as the cached result type -2. **Create reactive collection** for CMT → file_data mapping -3. **Implement delta processing** that only reprocesses changed files - -### Phase 3: Analysis Service - -1. **File watching integration** (can use fswatch, inotify, or external watcher) -2. **Service loop** that waits for events and re-runs analysis -3. **LSP integration** (optional) for editor support - -### Phase 4: Incremental Merge & Solve (Future) - -The current merge and solve phases are relatively fast (22% of time), but could be made incremental in the future: - -- Track which declarations changed -- Incrementally update reference graph -- Re-solve only affected transitive closure - -## Prototype Implementation - -Here's a minimal prototype showing how to integrate `Reactive_file_collection`: - -```ocaml -(* reactive_analysis.ml *) - -module CmtCollection = struct - type file_data = DceFileProcessing.file_data - - let collection : file_data Reactive_file_collection.t option ref = ref None - - let init ~config ~cmt_paths = - let coll = Reactive_file_collection.create - ~process:(fun (cmt_infos : Cmt_format.cmt_infos) -> - (* Extract file context from cmt_infos *) - let source_path = - match cmt_infos.cmt_annots |> FindSourceFile.cmt with - | Some path -> path - | None -> failwith "No source file" - in - let module_name = Paths.getModuleName source_path in - let is_interface = match cmt_infos.cmt_annots with - | Cmt_format.Interface _ -> true - | _ -> false - in - let file : DceFileProcessing.file_context = { - source_path; module_name; is_interface - } in - let cmtFilePath = "" (* not used in process_cmt_file body *) in - DceFileProcessing.process_cmt_file ~config ~file ~cmtFilePath cmt_infos - ) - in - (* Initial load *) - List.iter (Reactive_file_collection.add coll) cmt_paths; - collection := Some coll; - coll - - let apply_events events = - match !collection with - | Some coll -> Reactive_file_collection.apply coll events - | None -> failwith "Collection not initialized" - - let get_all_file_data () = - match !collection with - | Some coll -> Reactive_file_collection.values coll - | None -> [] -end - -(* Modified Reanalyze.runAnalysis *) -let runAnalysisIncremental ~config ~events = - (* Apply only the changed files *) - CmtCollection.apply_events events; - - (* Get all file_data (instant - values already computed) *) - let file_data_list = CmtCollection.get_all_file_data () in - - (* Rest of analysis is same as before *) - let annotations, decls, cross_file, refs, file_deps = - merge_all_builders file_data_list - in - solve ~annotations ~decls ~refs ~file_deps ~config -``` - -## Testing Strategy - -1. **Correctness**: Verify reactive analysis produces same results as batch -2. **Performance**: Benchmark incremental updates vs full analysis -3. **Edge cases**: - - File deletion during analysis - - Rapid successive changes - - Build errors (incomplete CMT files) - -## Open Questions - -1. **Build system integration**: How to get file events from rewatch? -2. **CMT staleness**: What if build system is still writing CMT files? -3. **Multi-project**: How to handle monorepos with multiple rescript.json? -4. **Memory limits**: When to evict file_data from cache? - -## Integration Points - -### 1. Shared.tryReadCmt → Marshal_cache - -Current code in `analysis/src/Shared.ml`: -```ocaml -let tryReadCmt cmt = - if not (Files.exists cmt) then ( - Log.log ("Cmt file does not exist " ^ cmt); - None) - else - match Cmt_format.read_cmt cmt with - | exception ... -> None - | x -> Some x -``` - -With Marshal_cache: -```ocaml -let tryReadCmt cmt = - if not (Files.exists cmt) then ( - Log.log ("Cmt file does not exist " ^ cmt); - None) - else - try - Some (Marshal_cache.with_unmarshalled_file cmt Fun.id) - with Marshal_cache.Cache_error (_, msg) -> - Log.log ("Invalid cmt format " ^ cmt ^ ": " ^ msg); - None -``` - -### 2. Reanalyze.loadCmtFile → Reactive_file_collection - -Current code in `analysis/reanalyze/src/Reanalyze.ml`: -```ocaml -let loadCmtFile ~config cmtFilePath : cmt_file_result option = - let cmt_infos = Cmt_format.read_cmt cmtFilePath in - ... -``` - -With reactive collection: -```ocaml -(* Global reactive collection *) -let cmt_collection : cmt_file_result Reactive_file_collection.t option ref = ref None - -let init_collection ~config = - cmt_collection := Some (Reactive_file_collection.create - ~process:(fun (cmt_infos : Cmt_format.cmt_infos) -> - process_cmt_infos ~config cmt_infos - )) - -let loadCmtFile_reactive ~config cmtFilePath = - match !cmt_collection with - | Some coll -> Reactive_file_collection.get coll cmtFilePath - | None -> loadCmtFile ~config cmtFilePath (* fallback *) -``` - -### 3. File Watcher Integration - -The analysis server already has `DceCommand.ml`. We can extend it to a service: - -```ocaml -(* DceService.ml *) - -type t = { - config: Reanalyze.DceConfig.t; - collection: cmt_file_result Reactive_file_collection.t; - mutable last_result: Reanalyze.AnalysisResult.t option; -} - -let create ~project_root = - let config = Reanalyze.DceConfig.current () in - let cmt_paths = Reanalyze.collectCmtFilePaths ~cmtRoot:None in - let collection = Reactive_file_collection.create - ~process:(process_cmt_for_config ~config) - in - List.iter (Reactive_file_collection.add collection) cmt_paths; - { config; collection; last_result = None } - -let on_file_change t events = - Reactive_file_collection.apply t.collection events; - (* Invalidate cached result *) - t.last_result <- None - -let analyze t = - match t.last_result with - | Some result -> result (* Cached, no files changed *) - | None -> - let file_data_list = Reactive_file_collection.values t.collection in - let result = run_analysis_on_file_data ~config:t.config file_data_list in - t.last_result <- Some result; - result -``` - -### 4. Build System Integration (rewatch) - -Rewatch already watches for file changes. We can extend it to notify the analysis service: - -In `rewatch/src/watcher.rs`: -```rust -// After successful compilation of a module -if let Some(analysis_socket) = &state.analysis_socket { - analysis_socket.send(AnalysisEvent::Modified(cmt_path)); -} -``` - -Or via a Unix domain socket/named pipe that the analysis service listens on. - -## Dependency Setup - -Add to `analysis/dune`: -```dune -(library - (name analysis) - (libraries - ... - skip-lite.marshal_cache - skip-lite.reactive_file_collection)) -``` - -Add to `analysis.opam`: -```opam -depends: [ - ... - "skip-lite" {>= "0.1"} -] -``` - -## Conclusion - -Integrating skip-lite's reactive collections with reanalyze offers a path to **39-390x speedup** for incremental analysis. The key insight is that CMT file loading (77% of current time) can be eliminated for unchanged files, and the processed file_data can be cached. - -The implementation requires: -1. Adding skip-lite as a dependency -2. Wrapping CMT loading with Marshal_cache (immediate benefit: mmap caching) -3. Creating reactive collection for file_data (benefit: only process changed files) -4. Creating a service mode that watches for file changes (benefit: persistent state) - -The merge and solve phases (23% of time) remain unchanged initially, but could be made incremental in the future for even greater speedups. - -## Next Steps - -1. **Phase 0**: Add skip-lite as optional dependency (behind a feature flag) -2. **Phase 1**: Replace `Cmt_format.read_cmt` with `Marshal_cache` wrapper -3. **Phase 2**: Benchmark improvement from mmap caching alone -4. **Phase 3**: Implement `Reactive_file_collection` for file_data -5. **Phase 4**: Create analysis service with file watching -6. **Phase 5**: Integrate with rewatch for automatic updates - diff --git a/packages/@rescript/runtime/Primitive_exceptions.res b/packages/@rescript/runtime/Primitive_exceptions.res index 22d8384cad6..f657ebb0133 100644 --- a/packages/@rescript/runtime/Primitive_exceptions.res +++ b/packages/@rescript/runtime/Primitive_exceptions.res @@ -40,7 +40,11 @@ type js_error = {cause: exn} | None -> (* assert it is not an exception *) ]} - This is not a problem in `try .. with` since the logic above is not expressible, see more design in [destruct_exn.md] + This is not a problem in `try .. with`: a handler only asks whether the + caught value matches an exception branch, never whether an arbitrary value + is an exception - the question a general exception-destruction operator + would force, and which cannot be answered soundly while open variants + share the exception representation. */ let isExtension = (type a, e: a): bool => if Primitive_js_extern.testAny(e) { diff --git a/rewatch/CompilerConfigurationSpec.md b/rewatch/CompilerConfigurationSpec.md index 3b21be0d338..13af9204d94 100644 --- a/rewatch/CompilerConfigurationSpec.md +++ b/rewatch/CompilerConfigurationSpec.md @@ -1,6 +1,10 @@ ## ReScript build configuration -This document contains a list of all config parameters with remarks, and whether they are already implemented in rewatch. It is based on https://rescript-lang.org/docs/manual/latest/build-configuration-schema. +This document lists configuration fields recognized by Rewatch and whether +their behavior is implemented. The public schema is maintained in the +[ReScript manual](https://rescript-lang.org/docs/manual/build-configuration-schema/). +Compatibility fields can remain implemented here after they stop being +recommended for new projects. | Parameter | JSON type | Remark | Implemented? | | --------------------- | ----------------------- | ----------------------------------------------------------- | :----------: | diff --git a/rewatch/README.md b/rewatch/README.md new file mode 100644 index 00000000000..1457fc1d074 --- /dev/null +++ b/rewatch/README.md @@ -0,0 +1,59 @@ +# Rewatch build system + +Rewatch is the Rust implementation of the ReScript build and watch commands. +It reads `rescript.json`, discovers packages and source files, maintains module +dependencies, and invokes the compiler in dependency order. + +## Code map + +- [`src/config.rs`](src/config.rs) parses configuration and converts supported + settings to compiler arguments. +- [`src/build/packages.rs`](src/build/packages.rs) discovers packages and + resolves package dependencies. +- [`src/build/parse.rs`](src/build/parse.rs) produces compiler AST artifacts. +- [`src/build/deps.rs`](src/build/deps.rs) builds the module dependency graph. +- [`src/build/compile.rs`](src/build/compile.rs) invokes the compiler and + updates build artifacts. +- [`src/watcher.rs`](src/watcher.rs) maps filesystem and configuration changes + to incremental rebuilds. +- [`src/cli.rs`](src/cli.rs) and [`src/main.rs`](src/main.rs) own command-line + parsing and dispatch. + +Focused documentation: + +- [configuration support matrix](CompilerConfigurationSpec.md) +- [monorepo discovery and build scope](MonorepoSupport.md) +- [feature-gated source directories](Features.md) +- [integration-test workspace](testrepo/README.md) + +The ReScript website owns user-facing configuration documentation. The support +matrix in this directory records what the current Rewatch implementation +accepts, including compatibility fields which are not recommended for new +projects. + +## Building and testing + +Run commands from the repository root: + +```sh +make rewatch +make test-rewatch +cargo test --manifest-path rewatch/Cargo.toml +cargo clippy --manifest-path rewatch/Cargo.toml --all-targets --all-features +cargo fmt --check --manifest-path rewatch/Cargo.toml +``` + +The integration suite is [`tests/suite.sh`](tests/suite.sh). It creates and +modifies projects under `testrepo/`; use its helpers for portable path and file +operations rather than adding platform-specific `sed` commands or fixed +sleeps. + +When a CLI or configuration value affects compilation, trace it through both +build and watch entry points. Command-line values override project +configuration. Keep that precedence and the compiler-argument conversion in +one owning layer where possible, and add both unit and integration coverage. + +Put public Rust API contracts in source documentation, local algorithm or +state invariants beside their implementation, and cross-module navigation in +this guide. Update or remove focused documents when the implementation changes; +do not preserve completed plans as descriptions of current behavior. diff --git a/tests/build_tests/super_errors/expected/object_coercion_readonly_to_mutable.res.expected b/tests/build_tests/super_errors/expected/object_coercion_readonly_to_mutable.res.expected index 60ebf93ff43..4cdef139848 100644 --- a/tests/build_tests/super_errors/expected/object_coercion_readonly_to_mutable.res.expected +++ b/tests/build_tests/super_errors/expected/object_coercion_readonly_to_mutable.res.expected @@ -2,7 +2,7 @@ We've found a bug for you! /.../fixtures/object_coercion_readonly_to_mutable.res:6:20-39 - 4 │ See docs/object_representation_cleanup.md. */ + 4 │ that restriction. */ 5 │ type t = {"x": int} 6 │ let p = (v: t) => (v :> {@set "x": int}) 7 │ diff --git a/tests/build_tests/super_errors/expected/object_write_closed_row.res.expected b/tests/build_tests/super_errors/expected/object_write_closed_row.res.expected index bf2ef07a031..d64ba49e7b1 100644 --- a/tests/build_tests/super_errors/expected/object_write_closed_row.res.expected +++ b/tests/build_tests/super_errors/expected/object_write_closed_row.res.expected @@ -2,8 +2,8 @@ We've found a bug for you! /.../fixtures/object_write_closed_row.res:6:28-37 - 4 │ See docs/object_representation_cleanup.md; compiling counterparts in - 5 │ tests/tests/src/object_mutability_pin.res. */ + 4 │ end-to-end object mutability tests, where assignment strengthens the + 5 │ function parameter instead. */ 6 │ let g = (o: {"x": int}) => o["x"] = 1 7 │ diff --git a/tests/build_tests/super_errors/fixtures/object_coercion_mutable_unequal.res b/tests/build_tests/super_errors/fixtures/object_coercion_mutable_unequal.res index 56eab519280..f76d8e33085 100644 --- a/tests/build_tests/super_errors/fixtures/object_coercion_mutable_unequal.res +++ b/tests/build_tests/super_errors/fixtures/object_coercion_mutable_unequal.res @@ -1,8 +1,8 @@ -/* Pin (object mutability cleanup): mutable-to-mutable coercion is invariant - in the field type — with unequal types it is rejected (today the setter - member demands contravariance while the getter demands covariance). Must - stay an error under the new model (Mutable A <: Mutable B iff A = B). - See docs/object_representation_cleanup.md. */ +/* Mutable fields are invariant in the field type. A mutable field of A is a + subtype of a mutable field of B only when A and B are equivalent: reads + require covariance, while writes require contravariance. This coercion + therefore fails because wide and narrow are not equivalent. */ + type wide = {"a": int, "b": int} type narrow = {"a": int} let p = (v: {@set "x": wide}) => (v :> {@set "x": narrow}) diff --git a/tests/build_tests/super_errors/fixtures/object_coercion_open_open.res b/tests/build_tests/super_errors/fixtures/object_coercion_open_open.res index 34824e62c64..376b862edb5 100644 --- a/tests/build_tests/super_errors/fixtures/object_coercion_open_open.res +++ b/tests/build_tests/super_errors/fixtures/object_coercion_open_open.res @@ -1,9 +1,9 @@ -/* Pin (object mutability cleanup): when BOTH rows are open, object fields - are invariant — this covariant coercion is rejected. Principled, not an - artifact: an open result is a promotable result, and a covariantly - weakened field must never remain promotable (a later write at the narrow - type would reach readers at the wide type). Must stay an error under the - new model. See docs/object_representation_cleanup.md. */ +/* Fields are invariant when both object rows are open. An open result remains + eligible for field promotion, so covariantly weakening its readable field + type would be unsafe: a later write at the narrow type could reach aliases + which read the field at the wide type. The coercion is therefore rejected. + */ + type wide = {"a": int, "b": int} type narrow = {"a": int} let p = (o: {.."x": wide}) => (o :> {.."x": narrow}) diff --git a/tests/build_tests/super_errors/fixtures/object_coercion_promote_readonly_caller.res b/tests/build_tests/super_errors/fixtures/object_coercion_promote_readonly_caller.res index 63579a6b8e7..744549e8e0b 100644 --- a/tests/build_tests/super_errors/fixtures/object_coercion_promote_readonly_caller.res +++ b/tests/build_tests/super_errors/fixtures/object_coercion_promote_readonly_caller.res @@ -1,10 +1,10 @@ -/* Pin (object mutability cleanup): COERCION-driven strengthening (as - opposed to the assignment-driven case in - object_open_write_readonly_caller.res): coercing an open-row parameter to - a same-type mutable target constrains the row, so a read-only caller is - rejected. Both halves must survive the new model (the coercion promotes - the open source's field; the demand becomes a Mutable field). - See docs/object_representation_cleanup.md. */ +/* Coercing an open-row parameter to a mutable target of the same field type + promotes the source field. That promotion is visible in the function's + parameter type, which now requires a mutable field from callers. A closed + read-only object cannot satisfy the strengthened parameter. This fixture + checks promotion through coercion rather than direct assignment, and + verifies that the strengthened requirement reaches the caller. */ + type wide = {"a": int, "b": int} let f = (o: {.."x": wide}) => (o :> {@set "x": wide}) @val external readonly: {"x": wide} = "readonly" diff --git a/tests/build_tests/super_errors/fixtures/object_coercion_readonly_to_mutable.res b/tests/build_tests/super_errors/fixtures/object_coercion_readonly_to_mutable.res index 451b43ba014..9cbf13a8257 100644 --- a/tests/build_tests/super_errors/fixtures/object_coercion_readonly_to_mutable.res +++ b/tests/build_tests/super_errors/fixtures/object_coercion_readonly_to_mutable.res @@ -1,6 +1,6 @@ -/* Pin (object mutability cleanup): a closed read-only field cannot be - coerced to a settable one — write capability cannot be conjured. Must - stay an error under the new model (closed row: no promotion). - See docs/object_representation_cleanup.md. */ +/* A closed read-only field cannot be coerced to a mutable field. Promotion is + available only while the object row is open, so this coercion cannot add + write capability to the source type. The target annotation does not change + that restriction. */ type t = {"x": int} let p = (v: t) => (v :> {@set "x": int}) diff --git a/tests/build_tests/super_errors/fixtures/object_open_write_readonly_caller.res b/tests/build_tests/super_errors/fixtures/object_open_write_readonly_caller.res index 53ceb57788f..df31eeb233f 100644 --- a/tests/build_tests/super_errors/fixtures/object_open_write_readonly_caller.res +++ b/tests/build_tests/super_errors/fixtures/object_open_write_readonly_caller.res @@ -1,8 +1,8 @@ -/* Pin (object mutability cleanup): writing a bare field of an OPEN row is - accepted but strengthens the function's demand — callers must supply a - writable field, so a read-only argument is rejected. Both halves must - survive the new model (write = promotion on the open row; the demand - becomes a Mutable field). See docs/object_representation_cleanup.md. */ +/* Writing a field in an open row promotes it to mutable. The promotion is + visible in the function's parameter type, which requires callers to supply + a mutable field. A closed read-only object cannot satisfy that strengthened + parameter. */ + let f = (o: {.."x": int}) => o["x"] = 1 @val external readonly: {"x": int} = "readonly" let _ = f(readonly) diff --git a/tests/build_tests/super_errors/fixtures/object_write_after_open_target_coercion.res b/tests/build_tests/super_errors/fixtures/object_write_after_open_target_coercion.res index 836ae8bee61..377db39dcc4 100644 --- a/tests/build_tests/super_errors/fixtures/object_write_after_open_target_coercion.res +++ b/tests/build_tests/super_errors/fixtures/object_write_after_open_target_coercion.res @@ -1,10 +1,10 @@ -/* Pin (object mutability cleanup): coercing a CLOSED source to an open - target yields a result whose row tail is instantiated from the source, - i.e. closed — so a subsequent write is rejected (the error even prints - the result type as the closed {"x": narrow}). This is what makes the - covariant closed-source/open-target coercion sound: the result is not - promotable. Must stay an error under the new model. - See docs/object_representation_cleanup.md. */ +/* Coercing a closed source to an open target instantiates the target's row + tail from the source. The result is therefore closed despite the target's + written form, and it cannot acquire write capability. This property makes + covariance sound in the closed-source, open-target case: the narrowed + result cannot later be promoted and used to write through the source. + */ + type wide = {"a": int, "b": int} type narrow = {"a": int} let p = (v: {"x": wide}) => { diff --git a/tests/build_tests/super_errors/fixtures/object_write_closed_row.res b/tests/build_tests/super_errors/fixtures/object_write_closed_row.res index e3503b94036..1a75edf59d9 100644 --- a/tests/build_tests/super_errors/fixtures/object_write_closed_row.res +++ b/tests/build_tests/super_errors/fixtures/object_write_closed_row.res @@ -1,6 +1,6 @@ -/* Pin (object mutability cleanup): writing a bare field of a CLOSED object - row is an error — the row cannot acquire a setter. Must stay an error - under the new model (Immutable field in a closed row cannot be promoted). - See docs/object_representation_cleanup.md; compiling counterparts in - tests/tests/src/object_mutability_pin.res. */ +/* A closed immutable field cannot be promoted. Assignment may promote an + immutable field only while its object row remains open; a closed row cannot + acquire write capability. The compiling open-row case is covered in the + end-to-end object mutability tests, where assignment strengthens the + function parameter instead. */ let g = (o: {"x": int}) => o["x"] = 1 diff --git a/tests/tests/src/object_mutability_pin.res b/tests/tests/src/object_mutability_pin.res index 2946e392ce2..7f7e2f4eda3 100644 --- a/tests/tests/src/object_mutability_pin.res +++ b/tests/tests/src/object_mutability_pin.res @@ -1,14 +1,5 @@ -/* Pins the typing behavior of object-field mutability - (docs/object_representation_cleanup.md). - - Every case in this file must keep compiling. The two cases that the - cleanup intentionally flipped to errors (a setter acquired at a type - different from the getter, via coercion or assignment) are pinned as - super_errors fixtures instead: object_coercion_setter_narrower.res and - object_setter_type_mismatch.res. - - The rejecting counterparts are pinned in - tests/build_tests/super_errors/fixtures/object_*.res. */ +/* Accepted object-field mutability and subtyping cases. Rejecting + counterparts are under tests/build_tests/super_errors/fixtures/object_*.res. */ type wide = {"a": int, "b": int} type narrow = {"a": int} /* wide <: narrow (width subtyping) */ @@ -17,24 +8,20 @@ type narrow = {"a": int} /* wide <: narrow (width subtyping) */ (Mutable A :> Immutable B with A <: B.) */ let forget_write_covariant = (v: {@set "x": wide}): {"x": narrow} => (v :> {"x": narrow}) -/* Open source, closed immutable target: ordinary covariance. Sound forever: - the coerced alias is read-only, and a later promotion of the source - writes at the source's own field type. */ +/* Open source, closed immutable target: ordinary covariance. The result is + read-only, while later source promotion writes at the source field type. */ let open_source_covariant = (o: {.."x": wide}): {"x": narrow} => (o :> {"x": narrow}) /* Closed source, open target: covariant; the target's tail is instantiated from the (closed) source, so the result is not promotable. */ let closed_source_open_target = (v: {"x": wide}) => (v :> {.."x": narrow}) -/* Open source, mutable target at the SAME type: accepted, and constrains - callers to writable objects (today: absorbs the "x#=" member; new model: - promotion Immutable -> Mutable at the same type). */ +/* Open source, mutable target at the same type: promotion constrains callers + to objects with a mutable field. */ let open_source_promote_same_type = (o: {.."x": wide}): {@set "x": wide} => (o :> {@set "x": wide}) -/* Writing a bare field of an open row is accepted and strengthens the - demand on callers (today: adds "x#=" through the tail; new model: - promotion). The rejection of a read-only caller is pinned in - object_open_write_readonly_caller.res. */ +/* Writing a field of an open row promotes it and strengthens the demand on + callers. */ let open_row_write = (o: {.."x": int}) => o["x"] = 1 /* A generalized getter accepts both read-only and settable objects. */ diff --git a/tools/README.md b/tools/README.md index da681035519..ef0dd1e59ab 100644 --- a/tools/README.md +++ b/tools/README.md @@ -32,6 +32,11 @@ rescript-tools doc src/EntryPointLibFile.res > doc.json rescript-tools reanalyze --help ``` +## Contributor documentation + +- [Migration framework capabilities](src/migrate.md) +- [Reanalyze architecture](../analysis/reanalyze/README.md) + ## Decode JSON Add to `bs-dev-dependencies`: diff --git a/tools/src/migrate.md b/tools/src/migrate.md index 340432bcc00..85c949a868a 100644 --- a/tools/src/migrate.md +++ b/tools/src/migrate.md @@ -1,6 +1,6 @@ # Migration Framework – Current Capabilities -This document captures what the migration framework currently supports, based on `tools/src/migrate.ml` (and helpers in `tools/src/transforms.ml`, `compiler/ml/builtin_attributes.ml`, and `analysis/src/Cmt.ml`). +This document captures what the migration framework currently supports, based on `tools/src/migrate.ml` (and helpers in `tools/src/transforms.ml`, `compiler/ml/builtin_attributes.ml`, and `analysis/src/cmt.ml`). ## Inputs & Preconditions