From b1975b81e67a1606987adfec286b36456d7b1354 Mon Sep 17 00:00:00 2001 From: Mageshwaran Rajendran Date: Mon, 14 Sep 2026 09:58:50 -0400 Subject: [PATCH 1/3] Explored and proposed required changes --- explore_scala.md | 30 +++ .../add-scala-language-support/.openspec.yaml | 2 + .../add-scala-language-support/design.md | 233 ++++++++++++++++++ .../add-scala-language-support/proposal.md | 87 +++++++ .../specs/scala-language-support/spec.md | 143 +++++++++++ .../add-scala-language-support/tasks.md | 56 +++++ openspec/changes/archive/.gitkeep | 0 openspec/config.yaml | 32 +++ openspec/specs/.gitkeep | 0 9 files changed, 583 insertions(+) create mode 100644 explore_scala.md create mode 100644 openspec/changes/add-scala-language-support/.openspec.yaml create mode 100644 openspec/changes/add-scala-language-support/design.md create mode 100644 openspec/changes/add-scala-language-support/proposal.md create mode 100644 openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md create mode 100644 openspec/changes/add-scala-language-support/tasks.md create mode 100644 openspec/changes/archive/.gitkeep create mode 100644 openspec/config.yaml create mode 100644 openspec/specs/.gitkeep diff --git a/explore_scala.md b/explore_scala.md new file mode 100644 index 00000000..fe98fc2a --- /dev/null +++ b/explore_scala.md @@ -0,0 +1,30 @@ +# Explore Scala Language Support + +## Requirements Specification: Scala Language Support +**Issue Reference:** #105 +**Status:** Approved for Community Contribution +**Primary Objective:** Implement parsing and dependency analysis support for the Scala programming language within CodeWiki. + +### 1. Context & Background +CodeWiki currently supports Java, JavaScript, Python, Ruby, and Kotlin. Users have requested Scala support to unify their workflows. Because the core engine utilizes AST Tree-sitter—which natively supports Scala—the foundational groundwork for this integration already exists. Due to its JVM-based nature, the Scala implementation will closely mirror the existing Java and Kotlin integrations. +### 2. Functional Requirements +- The system must successfully identify and parse files with the `.scala` extension. +- The system must be able to generate an Abstract Syntax Tree (AST) for Scala codebases. +- The system must correctly analyze dependencies and call graphs for Scala services. +### 3. Technical Implementation Requirements +The implementation requires updates across dependency management, core analyzer logic, and extension routing. +**3.1 Dependency Management** +- Update `pyproject.toml` to include the `tree-sitter-scala` package. +**3.2 Core Analyzer Creation** +- **File:** Create a new analyzer module at `codewiki/src/be/dependency_analyzer/analyzers/scala.py`. +- **Logic:** The analyzer should handle the JVM/package model specific to Scala. +- **Reference:** Use the existing Kotlin and Java analyzers as architectural templates. +**3.3 File Extension Registration** The `.scala` file extension must be registered in the existing routing and parsing modules. Update the following files to include `.scala` handling (similar to existing `.kt` configurations): +- `utils/patterns.py` +- `ast_parser.py` +- `analysis/call_graph_analyzer.py` +- `analyzers/artifact.py` +- `prompt_template.py` +### 4. Testing Requirements +- **Unit Tests:** Create a dedicated test file (e.g., `tests/test_scala_analyzer.py`) modeled after `tests/test_ruby_analyzer.py`. +- **Test Data:** Include at least one sample Scala code snippet to validate that the AST parsing and dependency analysis function correctly end-to-end. \ No newline at end of file diff --git a/openspec/changes/add-scala-language-support/.openspec.yaml b/openspec/changes/add-scala-language-support/.openspec.yaml new file mode 100644 index 00000000..e8cda9e5 --- /dev/null +++ b/openspec/changes/add-scala-language-support/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-10 diff --git a/openspec/changes/add-scala-language-support/design.md b/openspec/changes/add-scala-language-support/design.md new file mode 100644 index 00000000..b24b360e --- /dev/null +++ b/openspec/changes/add-scala-language-support/design.md @@ -0,0 +1,233 @@ +## Context + +See proposal.md — Why. Two aspects of the current state shape this design. + +**Scala is already half-wired.** `patterns.py` lists `*.scala` among code-file globs and +maps `".scala": "scala"` in `CODE_EXTENSIONS`. Scala files are therefore already +discovered and tagged with a language, then dropped at two later points: + +``` + [CODE_EXTENSIONS] .scala -> "scala" already wired + | + [call_graph_analyzer._find_code_files] passes through + | + [analysis_service._filter_supported_languages] DROPPED (whitelist) + | + [call_graph_analyzer._analyze_file if/elif chain] SILENT NO-OP (no branch, + | else-warning commented out) + [analyzers/scala.py] does not exist +``` + +The whitelist is the hard gate. Until `"scala"` joins it, nothing else in this change +executes, and because the dispatch fallthrough logs nothing, the failure mode during +development is silence rather than an error. + +**Leaf-node selection is typed, and its type set is narrow.** `leaf_selection.py` +defines `OOP_TYPES = {"class", "interface", "struct"}`. This drives two things: which +components survive `filter_leaf_nodes`, and the `n_oop` count feeding the issue-#75 +heuristic that decides whether free functions carry a codebase's architecture. Component +types outside that set — `trait` (emitted by PHP), `object` (Kotlin), `enum` and +`record` (Java), `delegate` (C#), `type_alias` (C++) — are already excluded from leaf +selection today. + +That exclusion is nearly invisible for existing languages because their codebases are +class-dominant. Scala is the first supported language where the excluded types *are* the +architecture: a typical Scala repository is mostly traits, objects, and case classes. A +faithful analyzer emitting `"trait"` and `"object"` would produce a repository whose +traits and objects are filtered out of documentation *and* whose `n_oop` count is near +zero, misfiring the heuristic into documenting loose methods instead of architectural +units. + +## Goals / Non-Goals + +**Goals:** + +- Land Scala support without changing behavior for any already-supported language. +- Keep the whole change inside the 14-file footprint in proposal.md — Impact. +- Model Scala's distinctive constructs (traits, companion objects) so the dependency + graph is correct rather than merely populated. + +**Non-Goals:** + +- Widening `OOP_TYPES`, adding duplicate-ID detection, or re-enabling the dispatch + warning. See proposal.md — Non-Goals; each is pre-existing and gets its own change. + Note that the `OOP_TYPES` gap is broader than PHP traits and Kotlin objects: C++ + `type_alias` components are dropped from leaf selection too, even though `cpp.py` + extracts them deliberately on the grounds that aliases are real API surface. Whichever + change widens the set should cover all of these together. +- Semantic analysis beyond syntactic extraction: no implicit resolution, no type + inference, no macro expansion. +- Treating `.sbt` as an analyzable source language (see Decision 5). + +## Decisions + +### 1. Use the two-level `component_type` / `node_type` split + +`component_type` carries the coarse type the pipeline gates on; `node_type` and +`display_name` carry the faithful Scala construct: + +| Scala construct | `component_type` | `node_type` | `display_name` | +| --- | --- | --- | --- | +| `class` / `case class` | `class` | `class` | `class Foo` | +| `trait` | `interface` | `trait` | `trait Foo` | +| `object` / `package object` | `class` | `object` | `object Foo` | +| Scala 3 `enum` | `class` | `enum` | `enum Foo` | +| method | `method` | `method` | `method Foo.bar` | +| top-level definition | `function` | `function` | `function bar` | + +*Why:* the coarse type keeps traits and objects leaf-eligible and counted toward +`n_oop`, so the issue-#75 heuristic reads a Scala repository accurately — while +`node_type` preserves what the construct actually is. No shared code changes, and no +information is discarded. + +This is an established pattern, not a new one. `typescript.py:561-563` already splits +the two, deriving `component_type` from a coarse `type` and `node_type` from a finer +`include_functions` heuristic on existing repositories. That deserves a dedicated change +`component_type`. The `Node` model carries all three fields plus `get_display_name()`. + +Critically, nothing recomputes `component_type` from `node_type`, so a deliberate split +cannot be clobbered downstream: `ast_parser._determine_component_type` — which has its +own competing whitelist including `enum`, `record`, `annotation`, and `delegate` — is +dead code, defined at `ast_parser.py:148` and never called. + +*Alternative considered — remap onto the existing vocabulary and discard the construct* +(emit `component_type="interface"` for a trait with no faithful `node_type`). Rejected +once the split was found: it loses information for no benefit, since setting the extra +fields is free. + +*Alternative considered — emit faithful `"trait"`/`"object"` as `component_type` and +widen `OOP_TYPES`.* Semantically cleanest, and it would fix the latent PHP, Kotlin, and +C++ gaps as a side effect. Rejected for this change: widening a globally shared set +changes documentation output for several shipped languages and can flip the +generated today reads from the coarse type, which the deferred `display_name` wiring +with a regression audit, not a rider on a new-language change. + +*Alternative considered — emit faithful types as `component_type` and leave `OOP_TYPES` +alone.* Rejected: it would ship Scala support that produces near-empty documentation for +idiomatic Scala. + +*Scope caveat — `node_type` and `display_name` are currently inert.* Neither flows into +generated documentation today: `get_display_name()` is never called, the MCP component +export at `mcp/tools/analysis.py:375` emits `component_type`, and `node_type` is read +only for the `artifact_file` check at `prompt_template.py:429` and a `"method"` test in +`call_graph_analyzer.py`. So setting these fields preserves the construct in the graph +and the exported artifacts at no cost, but it does **not** by itself make the docs say +"trait" rather than "interface". Surfacing faithful labels in documentation means wiring +`display_name` through the prompt path — a small change, deliberately not in this scope. +This design leaves the seam clean for it, and for the deferred `OOP_TYPES` work. + +### 2. Suffix companion objects in the component identifier + +Component IDs follow `relpath::Name` / `relpath::Class.method`. In Scala a companion +object is a *top-level peer* sharing its class's name, so `class Buffer` and +`object Buffer` both claim `buffer.scala::Buffer`. Because components are stored by +plain dict assignment with no collision check, the later-parsed declaration silently +displaces the earlier one — the class disappears from the graph while its methods remain +parented to the surviving object's node. + +The object therefore takes a suffixed identifier: `buffer.scala::Buffer, mirroring +Scala's own JVM encoding of module classes. Members follow their owner +(`buffer.scala::Buffer.push`, `buffer.scala::Buffer$.apply`). + +*Why:* both declarations survive as distinct components with their own source ranges and +documentation, and every edge stays attributable to the declaration that produced it. +The ` convention is one Scala developers already recognize from stack traces. + +*Alternative considered — merge the companion's members into the class as static +members.* Closer to how developers think about companions, but it conflates two +disjoint source ranges into one component, so the generated documentation would show one +declaration's source text under a component holding the other's members. + +*Note:* this is a local encoding choice, not a fix for the general silent-overwrite +defect, which is deferred. + +### 3. Use the dedicated `tree-sitter-scala` package + +*Why:* every one of the ten existing analyzers imports its own `tree_sitter_` +module and constructs a `Parser` from it. Following that pattern keeps the new analyzer +reviewable against its siblings. + +*Alternative considered — `tree_sitter_language_pack`,* already a dependency and already +capable of Scala. Rejected as inconsistent with the established per-language pattern; +switching parser sourcing is a repo-wide decision, not a Scala one. + +Version 0.26.2 introduces no `tree-sitter` core constraint in practice: its only core +pin is the optional `core` extra (`tree-sitter~=0.22`), which the pinned 0.23.2 already +satisfies. Wheels cover every CI platform. The grammar covers Scala 2 and 3, confirmed +against its published node types, which include `trait_definition`, +`object_definition`, `package_object`, `enum_definition`, `given_definition`, and +`extension_definition`. + +### 4. Register the language at every gate, not just the analyzer + +The whitelists in `analysis_service.py` (both the filter and the reported +supported-language list), the CLI's `SUPPORTED_EXTENSIONS` and language-detection map, +and the MCP incremental `source_extensions` set are all independent allowlists. Each +needs `.scala`/`.sc` or `"scala"` added. The Ruby precedent touched all of them; the +original requirements write-up named only some. + +*Why it matters:* the analyzer being correct is not sufficient for the feature to work. +Missing `repo_validator.py` alone makes the CLI reject a pure-Scala repository outright. + +### 5. Treat `.sbt` as a build artifact, never as analyzable source + +`build.sbt` and `project/*.sbt` join the manifest and build classifications in +`artifact.py` alongside the existing `pom.xml`, `build.gradle`, and `build.gradle.kts` +entries. `.sbt` is *not* added to `CODE_EXTENSIONS`. + +*Why:* `.sbt` files are syntactically Scala but semantically build configuration. +Documenting them as application source would misrepresent the architecture, while +leaving them unclassified means an sbt project's build story goes undocumented — a +visible gap now that artifact-aware generation has landed. + +`project/*.scala` (sbt meta-build code) remains ordinary Scala source. Special-casing it +adds a path-shaped exception for modest benefit. + +## Risks / Trade-offs + +**The Kotlin analyzer is a structural template, not a copy-paste source.** Kotlin's +grammar uses `class_declaration` / `object_declaration` / `function_declaration`; Scala's +uses `class_definition` / `object_definition` / `function_definition`. Every `node.type` +comparison needs rewriting. → Treat Kotlin as the reference for *shape* (node +extraction, then relationship extraction, module-path derivation, ID construction) and +derive all node type names from the Scala grammar's own node types. + +**Scala 3 significant-indentation syntax may parse poorly.** This is the only unknown +that could invalidate the extraction approach. → Spike first: parse representative +Scala 2 and Scala 3 files and assert the tree contains no `ERROR` nodes before building +extraction on top. Sequenced as the first task. + +**Scala's expression-oriented style may produce noisy call graphs.** Heavy chaining, +higher-order functions, and for-comprehensions generate many call expressions against +standard-library targets. → Follow the precedent of the existing analyzers and filter +primitives and common built-ins, as Kotlin and PHP already do with their primitive sets; +assert the exclusion in tests. + +**`.sc` is not exclusive to Scala** — SuperCollider and Scilab also use it. → Accepted: +the file is parsed with the Scala grammar and a non-Scala file simply yields no +components, matching how the other analyzers behave on unparseable input. + +**Decision 1's coarse type is still a compromise.** A Scala trait carries +`component_type="interface"`, so any consumer reading only the coarse type sees an +interface. → Largely mitigated by the split: `node_type="trait"` and +`display_name="trait Foo"` keep the construct in the graph, so no information is lost +and a future consumer can render it faithfully. The residual issue is that documentation +would address. +`ruby.py:608-611` sets `node_type` and a `" "` `display_name` alongside +`subtype` (a TS type alias is `component_type="type"`, `node_type="type_alias"`). + +## Open Questions + +These can be answered during implementation without changing the specs, the approach, +or the task breakdown: + +- **Scala 3 `given` definitions.** Typeclass instances are architecturally meaningful, + but whether each is a *documentable component* is unclear. Leaning toward extracting + named givens as `"class"` and skipping anonymous ones. Requires a real Scala 3 + codebase to judge signal versus noise. +- **Scala 3 `extension` blocks.** The methods inside are plausibly `"method"` components, + but their owner is the extended type, which may live outside the repository. +- **`type` aliases.** The existing `"type_alias"` component type sits outside + `OOP_TYPES` and would be dropped; likely skip rather than mismap. +- **`project/build.properties`.** Whether the sbt version pin is worth classifying as a + build artifact, or is too granular to document. diff --git a/openspec/changes/add-scala-language-support/proposal.md b/openspec/changes/add-scala-language-support/proposal.md new file mode 100644 index 00000000..e5627a41 --- /dev/null +++ b/openspec/changes/add-scala-language-support/proposal.md @@ -0,0 +1,87 @@ +## Why + +CodeWiki supports ten languages but not Scala, and users have asked for it to unify +JVM workflows (issue #105). The groundwork is already partly in place — `.scala` is +registered in `CODE_EXTENSIONS` and the tree-sitter Scala grammar exists — but Scala +files are recognized, tagged `language: "scala"`, and then silently discarded before +any analyzer runs. Today a Scala repository produces no documentation and no error. + +## What Changes + +- Add a tree-sitter Scala analyzer (`analyzers/scala.py`) that extracts classes, + traits, objects, enums, and methods and emits call/inheritance relationships. +- Register `.scala` and `.sc` as analyzable source across discovery, dispatch, + validation, and incremental-change detection. +- Admit `"scala"` to the two language whitelists in `analysis_service.py`. This is the + hard gate: without it every other change in this proposal is unreachable code. +- Classify `build.sbt` and `project/*.sbt` as build manifests so sbt-based projects get + their build story documented alongside the existing Maven/Gradle handling. +- Add `tree-sitter-scala` to `pyproject.toml` and `requirements.txt`. +- Add `tests/test_scala_analyzer.py` covering extraction, relationships, and an + end-to-end `DependencyParser` pass. +- Update the supported-languages list in `README.md`. + +No breaking changes. Existing languages are untouched: Scala constructs are mapped onto +the current `component_type` vocabulary rather than widening the shared `OOP_TYPES` set +(see design.md). + +## Capabilities + +### New Capabilities +- `scala-language-support`: Discovery, parsing, component extraction, and dependency + analysis for Scala source files, plus classification of sbt build files. + +### Modified Capabilities + +None. `openspec/specs/` is currently empty, so no existing capability's requirements +change. + +## Impact + +**Dependencies.** Adds `tree-sitter-scala` (0.26.2). No `tree-sitter` core bump: its +only core pin is the optional `core` extra (`tree-sitter~=0.22`), which the pinned +0.23.2 already satisfies. abi3-cp39 wheels are published for macOS x86_64/arm64, +manylinux x86_64/aarch64, musllinux, and Windows amd64/arm64, so no source builds in +CI. The grammar covers both Scala 2 and Scala 3. + +**Affected code** — 14 files, matching the footprint of the Ruby precedent (PR #97, +commit `05c7576`): + +| File | Change | +| --- | --- | +| `codewiki/src/be/dependency_analyzer/analyzers/scala.py` | new analyzer | +| `codewiki/src/be/dependency_analyzer/analysis/analysis_service.py` | `"scala"` in both language whitelists | +| `codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py` | dispatch branch, `lang-scala` node class | +| `codewiki/src/be/dependency_analyzer/utils/patterns.py` | `.sc` mapping, Scala function patterns | +| `codewiki/src/be/dependency_analyzer/ast_parser.py` | strip `.scala`/`.sc` in module paths | +| `codewiki/src/be/dependency_analyzer/analyzers/artifact.py` | source exts, sbt manifests | +| `codewiki/src/be/prompt_template.py` | `.scala`/`.sc` fence-language mapping | +| `codewiki/cli/utils/repo_validator.py` | `SUPPORTED_EXTENSIONS` | +| `codewiki/cli/utils/validation.py` | `detect_languages` map | +| `codewiki/mcp/tools/analysis.py` | `source_extensions` for incremental detection | +| `pyproject.toml`, `requirements.txt` | dependency | +| `tests/test_scala_analyzer.py` | new tests | +| `README.md` | supported-languages list | + +The last six are absent from the original requirements write-up but were all touched +when Ruby was added. Three of them are user-visible: without `repo_validator.py` the +CLI rejects a pure-Scala repository as containing no supported code, without +`validation.py` Scala is missing from detected-language statistics, and without +`mcp/tools/analysis.py` edits to `.scala` files never trigger incremental +regeneration. + +## Non-Goals + +Three adjacent defects were found while scoping this change. All are pre-existing, none +are Scala-specific, and each is deferred to its own change: + +- **`OOP_TYPES` excludes `trait`, `object`, `enum`, and `record`.** PHP already emits + `"trait"` and Kotlin `"object"`, so both are already dropped from leaf-node selection. + Widening the set is the semantically correct fix but changes documentation output for + three shipped languages and needs a regression audit. +- **Duplicate component IDs overwrite silently.** `ast_parser.py:109` and the ten + `self.functions[func_id] = func` sites in `call_graph_analyzer.py` are plain dict + assignments with no collision detection. +- **Unsupported-language dispatch is silent.** The `else: logger.warning(...)` at + `call_graph_analyzer.py:257` is commented out, which is why Scala files currently + fail without a diagnostic. diff --git a/openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md b/openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md new file mode 100644 index 00000000..611f64a1 --- /dev/null +++ b/openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md @@ -0,0 +1,143 @@ +## Purpose + +Lets CodeWiki analyze and document Scala repositories on the same terms as the other +supported languages: recognizing Scala source files, extracting their architectural +components and dependency relationships, and classifying sbt build files. + +## ADDED Requirements + +### Requirement: Scala source file recognition + +The system SHALL treat files with the `.scala` and `.sc` extensions as analyzable +source code, for both Scala 2 and Scala 3 syntax. + +#### Scenario: Repository containing only Scala sources is accepted + +- **WHEN** a user requests documentation for a repository whose only source files are `.scala` files +- **THEN** the repository passes validation instead of being rejected for containing no supported code + +#### Scenario: Scala appears in detected language statistics + +- **WHEN** a repository containing Scala files is scanned for language composition +- **THEN** Scala is reported with a file count reflecting its `.scala` and `.sc` files + +#### Scenario: Scala 3 syntax is parsed + +- **WHEN** a source file uses Scala 3 constructs such as `enum`, `given`, `extension`, or significant-indentation blocks +- **THEN** the file is parsed without error and its top-level definitions are extracted + +### Requirement: Scala component extraction + +The system SHALL extract Scala classes, case classes, traits, objects, package objects, +enums, and their members as documentable components, each carrying its name, source +location, source text, parameters, and documentation comment when present. + +#### Scenario: Traits and objects are documentable architectural units + +- **WHEN** a Scala file declares a trait and a standalone object +- **THEN** both are extracted as components eligible for documentation, not discarded + +#### Scenario: Methods are attributed to their enclosing type + +- **WHEN** a method is declared inside a class, trait, or object +- **THEN** the resulting component identifies that enclosing type as its owner + +#### Scenario: Top-level definitions are extracted as functions + +- **WHEN** a Scala file declares a definition outside any class, trait, or object +- **THEN** it is extracted as a free function component + +#### Scenario: Documentation comments are captured + +- **WHEN** a Scala declaration is preceded by a Scaladoc or line comment +- **THEN** the component records that text as its documentation + +### Requirement: Companion declarations remain distinct + +A class or trait and its same-named companion object SHALL both be preserved as +separate components with distinct identifiers. Neither declaration may displace the +other. + +#### Scenario: Class and companion object both survive + +- **WHEN** a file declares both `class Buffer` and `object Buffer` +- **THEN** two distinct components exist, each retaining its own source range, documentation, and members + +#### Scenario: Members are attributed to the correct companion + +- **WHEN** `class Buffer` declares `push` and `object Buffer` declares `apply` +- **THEN** `push` is attributed to the class and `apply` to the object, not both to one of them + +### Requirement: Scala dependency and call relationships + +The system SHALL emit dependency relationships for Scala code covering inheritance, +trait mixins, constructor parameter and field types, instantiation, and method calls. +Relationships whose target is defined in the analyzed repository SHALL resolve to that +component; relationships to symbols outside the repository SHALL be retained as +unresolved logical names. + +#### Scenario: Inheritance produces a dependency edge + +- **WHEN** a class extends a base class defined in another file in the repository +- **THEN** a resolved dependency edge links the subclass to that base class + +#### Scenario: Trait mixins produce dependency edges + +- **WHEN** a class mixes in a trait defined in the repository +- **THEN** a resolved dependency edge links the class to that trait + +#### Scenario: Intra-file calls resolve to sibling components + +- **WHEN** a method calls another method on the same type +- **THEN** a resolved dependency edge links the two method components + +#### Scenario: External symbols stay unresolved + +- **WHEN** Scala code references a type or method from the standard library or a third-party dependency +- **THEN** the relationship is recorded as an unresolved logical name rather than a repository component + +#### Scenario: Standard-library noise is excluded + +- **WHEN** Scala code uses ubiquitous built-ins such as primitive types or common collection operations +- **THEN** those references do not appear as dependency edges + +### Requirement: sbt build files are classified as build artifacts + +The system SHALL classify `build.sbt` and `.sbt` files under a `project/` directory as +build manifests, so that sbt-based projects have their build and packaging story +documented alongside the existing Maven and Gradle handling. + +#### Scenario: build.sbt is treated as a manifest + +- **WHEN** a repository contains a `build.sbt` at its root +- **THEN** that file is classified as a build manifest artifact rather than ignored or documented as application source + +#### Scenario: sbt plugin definitions are treated as build files + +- **WHEN** a repository contains `project/plugins.sbt` +- **THEN** that file is classified as a build artifact + +### Requirement: End-to-end documentation generation for Scala repositories + +A Scala repository SHALL produce documentation describing its actual components. Scala +files may not be silently dropped between discovery and analysis. + +#### Scenario: Scala components reach the generated documentation + +- **WHEN** documentation is generated for a repository of Scala source files +- **THEN** the generated output describes components extracted from those files + +#### Scenario: Scala code snippets are tagged for highlighting + +- **WHEN** generated documentation embeds a code snippet taken from a Scala source file +- **THEN** the snippet is tagged as Scala so it renders with Scala syntax highlighting + +### Requirement: Incremental regeneration reacts to Scala edits + +The system SHALL detect modifications to Scala source files when determining whether a +repository has changed since its last documentation run. + +#### Scenario: A modified Scala file marks the repository as changed + +- **WHEN** a `.scala` file is modified after the previous documentation run +- **THEN** the change detection reports that file as changed diff --git a/openspec/changes/add-scala-language-support/tasks.md b/openspec/changes/add-scala-language-support/tasks.md new file mode 100644 index 00000000..a1572034 --- /dev/null +++ b/openspec/changes/add-scala-language-support/tasks.md @@ -0,0 +1,56 @@ +## 1. Grammar Spike + +- [ ] 1.1 Add `tree-sitter-scala>=0.26.2` to `pyproject.toml` and pin `tree-sitter-scala==0.26.2` in `requirements.txt`; verify `pip install -e .` succeeds and `python -c "import tree_sitter_scala"` imports without a source build +- [ ] 1.2 Parse a representative Scala 2 file and a Scala 3 file using significant-indentation syntax, `enum`, `given`, and `extension`; verify neither tree contains `ERROR` or `MISSING` nodes. If Scala 3 indentation fails, stop and revisit the design before continuing — this gates the extraction approach +- [ ] 1.3 Record the concrete grammar node type names for class, case class, trait, object, package object, enum, method, and top-level definitions; verify each name appears in the grammar's node types rather than being carried over from the Kotlin analyzer + +## 2. Analyzer Core + +- [ ] 2.1 Create `codewiki/src/be/dependency_analyzer/analyzers/scala.py` with a `TreeSitterScalaAnalyzer` class and an `analyze_scala_file(file_path, content, repo_path)` entry point mirroring the Kotlin analyzer's structure; verify it returns empty node and relationship lists for an empty file without raising +- [ ] 2.2 Implement module-path and relative-path derivation stripping `.scala` and `.sc`; verify a nested file yields a dotted module path with the extension removed +- [ ] 2.3 Implement component ID construction, giving companion objects the `suffix per design Decision 2; verify a file declaring both` class Buffer `and` object Buffer `yields` buffer.scala::Buffer `and` buffer.scala::Buffer as distinct components with neither displaced +- [ ] 2.4 Extract classes, case classes, traits, objects, package objects, and Scala 3 enums, applying the type mapping from design Decision 1; verify a trait is emitted with component type `interface` and an object with `class` +- [ ] 2.5 Extract methods with their enclosing type as owner, and top-level definitions as free functions; verify a method inside a trait is attributed to that trait and a top-level `def` is emitted as a function +- [ ] 2.6 Extract parameters, source ranges, and Scaladoc or line-comment documentation for each component; verify a documented method reports its parameter list and a non-empty docstring + +## 3. Relationship Extraction + +- [ ] 3.1 Emit inheritance and trait-mixin edges from `extends`/`with` clauses; verify a class extending a repository-local base class and mixing in a repository-local trait produces a resolved edge for each +- [ ] 3.2 Emit edges for constructor parameter types, field types, and instantiation; verify a class with a field of a repository-local type produces a resolved edge to it +- [ ] 3.3 Emit method-call edges, resolving intra-file targets to sibling components and leaving external targets as unresolved logical names; verify an intra-type call resolves and a standard-library call does not +- [ ] 3.4 Define a Scala primitive and common built-in exclusion set following the Kotlin and PHP precedent; verify references to primitive types and ubiquitous collection operations produce no edges + +## 4. Pipeline Registration + +- [ ] 4.1 Add `"scala"` to both language sets in `analysis/analysis_service.py` (`_filter_supported_languages` and `_get_supported_languages`); verify Scala files are no longer dropped before dispatch. This is the hard gate — without it tasks 2 and 3 are unreachable +- [ ] 4.2 Add the `scala` dispatch branch and an `_analyze_scala_file` method to `analysis/call_graph_analyzer.py`, plus the `lang-scala` node class alongside the existing `lang-kotlin` handling; verify a Scala file in a test repository produces components end to end +- [ ] 4.3 Map `.sc` to `"scala"` in `utils/patterns.py` `CODE_EXTENSIONS`, add `*.sc` to the code-file globs, and add Scala entries to `FUNCTION_DEFINITION_PATTERNS`; verify `.scala` and `.sc` both resolve to the `scala` language +- [ ] 4.4 Add `.scala` and `.sc` to the extension list in `ast_parser.py` `_file_to_module_path`; verify module paths for Scala files drop the extension +- [ ] 4.5 Add `.scala` and `.sc` to the source-extension set in `analyzers/artifact.py` so Scala files classify as code rather than falling through to artifact detection; verify a plain `.scala` file is not classified as an artifact +- [ ] 4.6 Add `.scala` and `.sc` to the fence-language map in `src/be/prompt_template.py`; verify a Scala snippet is tagged `scala` + +## 5. sbt Build File Classification + +- [ ] 5.1 Add `build.sbt` to `_MANIFEST_NAMES` and `.sbt` handling for `project/` files in `analyzers/artifact.py`; verify `build.sbt` classifies as a manifest and `project/plugins.sbt` as a build artifact +- [ ] 5.2 Confirm `.sbt` is absent from `CODE_EXTENSIONS` per design Decision 5; verify an `.sbt` file is never dispatched to the Scala analyzer + +## 6. CLI and MCP Gates + +- [ ] 6.1 Add `.scala` and `.sc` to `SUPPORTED_EXTENSIONS` in `cli/utils/repo_validator.py`; verify a repository containing only Scala files passes validation instead of being rejected as having no supported code +- [ ] 6.2 Add a `"Scala": [".scala", ".sc"]` entry to `detect_languages` in `cli/utils/validation.py`; verify Scala appears with a file count in detected language statistics +- [ ] 6.3 Add `.scala` and `.sc` to `source_extensions` in `mcp/tools/analysis.py`; verify a modified `.scala` file is reported as changed by incremental detection + +## 7. Tests + +- [ ] 7.1 Create `tests/test_scala_analyzer.py` modeled on `tests/test_ruby_analyzer.py`, with `pytest.importorskip("tree_sitter_scala")` and a Scala sample exercising a trait, a class with a companion object, inheritance, a mixin, an intra-type call, and a top-level definition; verify the suite runs +- [ ] 7.2 Add extraction assertions covering component types, owners, IDs, and the companion ` suffix; verify all pass +- [ ] 7.3 Add relationship assertions covering resolved inheritance, resolved mixin, resolved intra-file call, unresolved external target, and excluded built-in noise; verify all pass +- [ ] 7.4 Add a `DependencyParser` end-to-end test across two Scala files asserting cross-file inheritance resolves into `depends_on`; verify it passes +- [ ] 7.5 Add a Scala 3 sample covering `enum` and significant-indentation syntax; verify components are extracted +- [ ] 7.6 Add an artifact-classification test for `build.sbt` and `project/plugins.sbt`; verify both classify as build artifacts +- [ ] 7.7 Run the full existing test suite; verify no regressions in the Python, Kotlin, PHP, Ruby, or artifact analyzer tests + +## 8. Documentation + +- [ ] 8.1 Add Scala to the supported-languages list in `README.md`; verify it renders alongside the existing ten entries +- [ ] 8.2 Resolve the design's open questions on `given`, `extension`, `type` aliases, and `project/build.properties` against a real Scala 3 repository, and record the outcomes in `design.md`; verify each open question is either answered or explicitly carried forward diff --git a/openspec/changes/archive/.gitkeep b/openspec/changes/archive/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/openspec/config.yaml b/openspec/config.yaml new file mode 100644 index 00000000..c4d34ace --- /dev/null +++ b/openspec/config.yaml @@ -0,0 +1,32 @@ +schema: spec-driven + +# Project context (optional) +# This is shown to AI when creating artifacts. +# Add your tech stack, conventions, style guides, domain knowledge, etc. +# Example: +# context: | +# Tech stack: TypeScript, React, Node.js +# We use conventional commits +# Domain: e-commerce platform + +# Per-artifact rules (optional) +# Add custom rules for specific artifacts. +# Example: +# rules: +# proposal: +# - Keep proposals under 500 words +# - Always include a "Non-goals" section +# tasks: +# - Break tasks into chunks of max 2 hours + +# Per-operation guidance (optional) +# Add advisory guidance for how apply and archive work should be conducted. +# This is separate from artifact rules above. +# Example: +# operations: +# apply: +# guidance: +# - Keep test summaries concise +# archive: +# guidance: +# - Summarize the archive outcome before finishing diff --git a/openspec/specs/.gitkeep b/openspec/specs/.gitkeep new file mode 100644 index 00000000..e69de29b From 89ca2b7b82c0ecdcc8eab46f64458185cc9531f3 Mon Sep 17 00:00:00 2001 From: Mageshwaran Rajendran Date: Mon, 14 Sep 2026 12:57:46 -0400 Subject: [PATCH 2/3] Adding support for scala language analyzer --- .gitignore | 1 + README.md | 2 +- codewiki/cli/utils/repo_validator.py | 2 + codewiki/cli/utils/validation.py | 1 + codewiki/mcp/tools/analysis.py | 2 + .../analysis/analysis_service.py | 2 + .../analysis/call_graph_analyzer.py | 26 + .../dependency_analyzer/analyzers/artifact.py | 5 +- .../be/dependency_analyzer/analyzers/scala.py | 680 ++++++++++++++++++ .../src/be/dependency_analyzer/ast_parser.py | 2 + .../be/dependency_analyzer/utils/patterns.py | 10 + codewiki/src/be/prompt_template.py | 2 + .../add-scala-language-support/.openspec.yaml | 2 - .../add-scala-language-support/design.md | 233 ------ .../add-scala-language-support/proposal.md | 87 --- .../specs/scala-language-support/spec.md | 143 ---- .../add-scala-language-support/tasks.md | 56 -- openspec/changes/archive/.gitkeep | 0 openspec/config.yaml | 32 - openspec/specs/.gitkeep | 0 pyproject.toml | 1 + requirements.txt | 1 + tests/test_scala_analyzer.py | 254 +++++++ 23 files changed, 989 insertions(+), 555 deletions(-) create mode 100644 codewiki/src/be/dependency_analyzer/analyzers/scala.py delete mode 100644 openspec/changes/add-scala-language-support/.openspec.yaml delete mode 100644 openspec/changes/add-scala-language-support/design.md delete mode 100644 openspec/changes/add-scala-language-support/proposal.md delete mode 100644 openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md delete mode 100644 openspec/changes/add-scala-language-support/tasks.md delete mode 100644 openspec/changes/archive/.gitkeep delete mode 100644 openspec/config.yaml delete mode 100644 openspec/specs/.gitkeep create mode 100644 tests/test_scala_analyzer.py diff --git a/.gitignore b/.gitignore index b6cd8465..34bf8571 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,7 @@ tests/* !tests/test_processing_order_update.py !tests/test_leaf_selection.py !tests/test_artifact_analyzer.py +!tests/test_scala_analyzer.py # Jupyter *.ipynb diff --git a/README.md b/README.md index ea2fc808..e281f7ba 100644 --- a/README.md +++ b/README.md @@ -149,7 +149,7 @@ CodeWiki is an open-source framework for **automated repository-level documentat ### Supported Languages -**🐍 Python** • **☕ Java** • **🟨 JavaScript** • **🔷 TypeScript** • **⚙️ C** • **🔧 C++** • **🪟 C#** • **🎯 Kotlin** • **🐘 PHP** • **💎 Ruby** +**🐍 Python** • **☕ Java** • **🟨 JavaScript** • **🔷 TypeScript** • **⚙️ C** • **🔧 C++** • **🪟 C#** • **🎯 Kotlin** • **🐘 PHP** • **💎 Ruby** • **🔺 Scala** --- diff --git a/codewiki/cli/utils/repo_validator.py b/codewiki/cli/utils/repo_validator.py index 46a5fcfe..53ccf253 100644 --- a/codewiki/cli/utils/repo_validator.py +++ b/codewiki/cli/utils/repo_validator.py @@ -31,6 +31,8 @@ ".kt", # Kotlin ".kts", # Kotlin Scripts ".rb", # Ruby + ".scala", # Scala + ".sc", # Scala Scripts } diff --git a/codewiki/cli/utils/validation.py b/codewiki/cli/utils/validation.py index 5056be7c..13dda9be 100644 --- a/codewiki/cli/utils/validation.py +++ b/codewiki/cli/utils/validation.py @@ -166,6 +166,7 @@ def detect_supported_languages(directory: Path) -> list[tuple[str, int]]: "PHP": [".php", ".phtml", ".inc"], "Kotlin": [".kt", ".kts"], "Ruby": [".rb"], + "Scala": [".scala", ".sc"], } # Directories to exclude from counting diff --git a/codewiki/mcp/tools/analysis.py b/codewiki/mcp/tools/analysis.py index a08623eb..f1ed0b6e 100644 --- a/codewiki/mcp/tools/analysis.py +++ b/codewiki/mcp/tools/analysis.py @@ -226,6 +226,8 @@ def _detect_via_mtime( ".kt", ".kts", ".rb", + ".scala", + ".sc", } changed: list[str] = [] diff --git a/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py b/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py index f0cb468d..5ac2a19f 100644 --- a/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py +++ b/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py @@ -354,6 +354,7 @@ def _filter_supported_languages(self, code_files: list[dict]) -> list[dict]: "go", "rust", "kotlin", + "scala", } return [ @@ -375,6 +376,7 @@ def _get_supported_languages(self) -> list[str]: "php", "ruby", "kotlin", + "scala", ] def _cleanup_repository(self, temp_dir: str): diff --git a/codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py b/codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py index dfcd7340..064438fa 100644 --- a/codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py +++ b/codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py @@ -253,6 +253,8 @@ def _analyze_code_file(self, repo_dir: str, file_info: dict): self._analyze_php_file(file_path, content, repo_dir) elif language == "ruby": self._analyze_ruby_file(file_path, content, repo_dir) + elif language == "scala": + self._analyze_scala_file(file_path, content, repo_dir) # else: # logger.warning( # f"Unsupported language for call graph analysis: {language} for file {file_path}" @@ -508,6 +510,28 @@ def _analyze_ruby_file(self, file_path: str, content: str, repo_dir: str): except Exception: logger.exception(f"Failed to analyze Ruby file {file_path}") + def _analyze_scala_file(self, file_path: str, content: str, repo_dir: str): + """ + Analyze Scala file using tree-sitter based analyzer. + + Args: + file_path: Relative path to the Scala file + content: File content string + repo_dir: Repository base directory + """ + from codewiki.src.be.dependency_analyzer.analyzers.scala import analyze_scala_file + + try: + functions, relationships = analyze_scala_file(file_path, content, repo_path=repo_dir) + + for func in functions: + func_id = func.id if func.id else f"{file_path}:{func.name}" + self.functions[func_id] = func + + self.call_relationships.extend(relationships) + except Exception: + logger.exception(f"Failed to analyze Scala file {file_path}") + def _resolve_call_relationships(self): """ Resolve function call relationships across all languages. @@ -804,6 +828,8 @@ def _generate_visualization_data(self) -> dict: node_classes.append("lang-php") elif file_ext == ".rb": node_classes.append("lang-ruby") + elif file_ext in [".scala", ".sc"]: + node_classes.append("lang-scala") cytoscape_elements.append( { diff --git a/codewiki/src/be/dependency_analyzer/analyzers/artifact.py b/codewiki/src/be/dependency_analyzer/analyzers/artifact.py index 7c2b175f..fdf0daff 100644 --- a/codewiki/src/be/dependency_analyzer/analyzers/artifact.py +++ b/codewiki/src/be/dependency_analyzer/analyzers/artifact.py @@ -241,6 +241,7 @@ def is_artifact_file_node(node: Any) -> bool: "build.gradle.kts", "settings.gradle", "settings.gradle.kts", + "build.sbt", "Package.swift", "pubspec.yaml", "Pipfile", @@ -293,7 +294,7 @@ def is_artifact_file_node(node: Any) -> bool: "Gruntfile.js", "Herebyfile.mjs", } -_BUILD_EXTS = {".gn", ".gni", ".gradle", ".rake", ".mk", ".cmake", ".bzl", ".ninja"} +_BUILD_EXTS = {".gn", ".gni", ".gradle", ".rake", ".mk", ".cmake", ".bzl", ".ninja", ".sbt"} _BUILD_CONFIG_RE = re.compile(r"^(webpack|rollup|vite|esbuild|tsup|babel)\.config\.[cm]?[jt]s$") _BUILD_TOPS = {"build", "rakelib", "cmake"} _TEST_INFRA_NAMES = { @@ -427,6 +428,8 @@ def classify_artifact( ".cs", ".php", ".kt", + ".scala", + ".sc", }: if name in _TEST_INFRA_NAMES: return "test_infra" diff --git a/codewiki/src/be/dependency_analyzer/analyzers/scala.py b/codewiki/src/be/dependency_analyzer/analyzers/scala.py new file mode 100644 index 00000000..3fa95e7c --- /dev/null +++ b/codewiki/src/be/dependency_analyzer/analyzers/scala.py @@ -0,0 +1,680 @@ +"""Tree-sitter based Scala analyzer. + +Extracts classes, case classes, traits, objects, package objects, Scala 3 +enums, and their methods as documentable components, plus inheritance, +mixin, field/constructor-type, instantiation, and call dependency edges. + +Component types follow the two-level `component_type` / `node_type` split +described in openspec/changes/add-scala-language-support/design.md — +Decision 1: `component_type` is the coarse type the pipeline gates leaf +selection on (traits map to "interface", objects to "class" so both stay +leaf-eligible), while `node_type` and `display_name` preserve the faithful +Scala construct. + +Companion objects get a `$`-suffixed component id (Decision 2), mirroring +Scala's own JVM encoding of module classes, so a class and its same-named +companion never collide. +""" + +import logging +import os +from pathlib import Path + +import tree_sitter_scala +from tree_sitter import Language, Parser + +from codewiki.src.be.dependency_analyzer.models.core import CallRelationship, Node + +logger = logging.getLogger(__name__) + +# Node types that introduce a new component scope (owner of members / target +# of extends-clause edges). +_TYPE_DEFINITION_NODES = ( + "class_definition", + "trait_definition", + "object_definition", + "package_object", + "enum_definition", +) +_METHOD_DEFINITION_NODES = ("function_definition", "function_declaration") + +# `component_type` / `node_type` / display-prefix per design.md Decision 1. +_TYPE_MAPPING = { + "trait_definition": ("interface", "trait"), + "class_definition": ("class", "class"), + "object_definition": ("class", "object"), + "package_object": ("class", "object"), + "enum_definition": ("class", "enum"), +} + +# Scala primitives and common built-in types excluded from dependency edges, +# following the Kotlin and PHP precedent (design.md — Risks/Trade-offs). +SCALA_PRIMITIVE_TYPES = frozenset( + { + "Int", + "Long", + "Short", + "Byte", + "Double", + "Float", + "Boolean", + "Char", + "String", + "Unit", + "Any", + "AnyRef", + "AnyVal", + "Nothing", + "Null", + "Object", + "List", + "Seq", + "Vector", + "Array", + "Set", + "Map", + "Option", + "Some", + "None", + "Either", + "Left", + "Right", + "Try", + "Success", + "Failure", + "Iterable", + "Iterator", + "Tuple2", + "Tuple3", + "Tuple4", + "Tuple5", + "Function0", + "Function1", + "Function2", + "PartialFunction", + } +) + +# Ubiquitous collection/standard-library method names that never point at a +# repository component. Calls whose only signal is one of these names are +# dropped instead of emitted as unresolved relationships. +SCALA_CORE_CALLS = frozenset( + { + "map", + "flatMap", + "filter", + "filterNot", + "foreach", + "fold", + "foldLeft", + "foldRight", + "reduce", + "reduceLeft", + "reduceRight", + "collect", + "collectFirst", + "sortBy", + "sortWith", + "sorted", + "groupBy", + "partition", + "zip", + "zipWithIndex", + "take", + "takeWhile", + "drop", + "dropWhile", + "head", + "headOption", + "tail", + "last", + "lastOption", + "isEmpty", + "nonEmpty", + "size", + "length", + "contains", + "exists", + "forall", + "find", + "sum", + "min", + "max", + "mkString", + "toList", + "toSeq", + "toSet", + "toMap", + "toVector", + "toArray", + "toString", + "apply", + "getOrElse", + "orElse", + "get", + "isDefined", + "flatten", + "distinct", + "reverse", + "append", + "prepend", + "copy", + "hashCode", + "equals", + "println", + "print", + "require", + "assert", + } +) + + +class TreeSitterScalaAnalyzer: + def __init__(self, file_path: str, content: str, repo_path: str | None = None): + self.file_path = Path(file_path) + self.content = content + self.repo_path = repo_path or "" + self.nodes: list[Node] = [] + self.call_relationships: list[CallRelationship] = [] + # Same-file symbol table keyed by logical name ("Foo", "Foo$", "Foo.bar"). + self.top_level_nodes: dict = {} + self.seen_relationships: set = set() + self._analyze() + + # ------------------------------------------------------------------ + # Setup + # ------------------------------------------------------------------ + + def _get_relative_path(self) -> str: + if self.repo_path: + try: + return os.path.relpath(str(self.file_path), self.repo_path) + except ValueError: + return str(self.file_path) + return str(self.file_path) + + def _get_component_id(self, logical_name: str) -> str: + return f"{self._get_relative_path()}::{logical_name}" + + def _analyze(self): + try: + scala_language = Language(tree_sitter_scala.language()) + parser = Parser(scala_language) + tree = parser.parse(bytes(self.content, "utf8")) + root = tree.root_node + lines = self.content.splitlines() + + self._extract_nodes(root, lines) + self._extract_relationships(root) + except Exception as e: # noqa: BLE001 — a broken file must not abort the sweep + logger.error(f"Error parsing Scala file {self.file_path}: {e}") + + @staticmethod + def _field_text(node, field_name: str) -> str | None: + child = node.child_by_field_name(field_name) + return child.text.decode() if child is not None else None + + # ------------------------------------------------------------------ + # Scope helpers + # ------------------------------------------------------------------ + + def _owner_key_for_definition(self, def_node) -> str | None: + """The logical name a class/trait/object/enum is registered under.""" + name = self._field_text(def_node, "name") + if not name: + return None + if def_node.type in ("object_definition", "package_object"): + return f"{name}$" + return name + + def _find_containing_type(self, node) -> str | None: + """Walk up to the nearest enclosing class/trait/object/enum and return + its owner key (`$`-suffixed for object-kind constructs), or None at + file scope.""" + current = node.parent + while current is not None: + if current.type in _TYPE_DEFINITION_NODES: + return self._owner_key_for_definition(current) + current = current.parent + return None + + def _find_containing_method(self, node) -> str | None: + """Walk up to the nearest enclosing method/function and return its + logical (owner-qualified) name, or None.""" + current = node.parent + while current is not None: + if current.type in _METHOD_DEFINITION_NODES: + name = self._field_text(current, "name") + if name: + owner = self._find_containing_type(current) + return f"{owner}.{name}" if owner else name + current = current.parent + return None + + def _resolve_caller(self, node) -> str | None: + """Prefer the enclosing method; fall back to the enclosing type for + expressions in field initializers.""" + method = self._find_containing_method(node) + if method: + return self._get_component_id(method) + owner = self._find_containing_type(node) + if owner: + return self._get_component_id(owner) + return None + + # ------------------------------------------------------------------ + # Pass 1: components + # ------------------------------------------------------------------ + + def _extract_nodes(self, node, lines): + component_type = None + node_type = None + logical_name = None + class_name = None + parameters = None + base_classes = None + + if node.type in _TYPE_DEFINITION_NODES: + logical_name = self._owner_key_for_definition(node) + if logical_name: + component_type, node_type = _TYPE_MAPPING[node.type] + base_classes = self._extract_extends_types(node) + elif node.type in _METHOD_DEFINITION_NODES: + name = self._field_text(node, "name") + if name: + parameters = self._extract_parameters(node) + owner = self._find_containing_type(node) + if owner: + component_type, node_type = "method", "method" + logical_name = f"{owner}.{name}" + class_name = owner + else: + component_type, node_type = "function", "function" + logical_name = name + + if component_type and logical_name: + self._add_node( + node, + logical_name, + component_type, + node_type, + lines, + class_name=class_name, + parameters=parameters, + base_classes=base_classes, + ) + + for child in node.children: + self._extract_nodes(child, lines) + + def _extract_parameters(self, func_node) -> list[str] | None: + # A curried method (`def add(x: Int)(y: Int)`) has one `parameters` + # node per group; child_by_field_name would silently return only the + # first, so every group is collected via children_by_field_name. + params_nodes = func_node.children_by_field_name("parameters") + params = [] + for params_node in params_nodes: + for child in params_node.children: + if child.type == "parameter": + params.append(child.text.decode().strip()) + return params or None + + def _add_node( + self, + node, + logical_name: str, + component_type: str, + node_type: str, + lines, + class_name: str | None = None, + parameters: list[str] | None = None, + base_classes: list[str] | None = None, + ): + component_id = self._get_component_id(logical_name) + relative_path = self._get_relative_path() + + docstring = "" + comment = node.prev_sibling + if comment is not None and comment.type in ("block_comment", "comment"): + docstring = comment.text.decode().strip() + + start_line_idx = node.start_point[0] + end_line_idx = node.end_point[0] + 1 + code_snippet = ( + "\n".join(lines[start_line_idx:end_line_idx]) if start_line_idx < len(lines) else "" + ) + + display_name = f"{node_type} {logical_name}" + + node_obj = Node( + id=component_id, + name=logical_name, + component_type=component_type, + file_path=str(self.file_path), + relative_path=relative_path, + source_code=code_snippet, + start_line=node.start_point[0] + 1, + end_line=node.end_point[0] + 1, + has_docstring=bool(docstring), + docstring=docstring, + parameters=parameters, + node_type=node_type, + base_classes=base_classes, + class_name=class_name, + display_name=display_name, + component_id=component_id, + language="scala", + ) + self.nodes.append(node_obj) + self.top_level_nodes[logical_name] = node_obj + + # ------------------------------------------------------------------ + # Pass 2: relationships + # ------------------------------------------------------------------ + + def _extract_relationships(self, node): + if node.type in _TYPE_DEFINITION_NODES: + self._emit_extends_edges(node) + elif node.type == "class_parameter": + self._emit_class_parameter_edge(node) + elif node.type in ("val_definition", "var_definition"): + self._emit_field_type_edge(node) + elif node.type == "instance_expression": + self._emit_instantiation_edge(node) + elif node.type == "call_expression": + self._emit_call_edge(node) + + for child in node.children: + self._extract_relationships(child) + + def _get_type_name(self, node) -> str | None: + """Get the primary type name from a type node, stripping generics.""" + if node is None: + return None + if node.type == "type_identifier": + return node.text.decode() + if node.type == "generic_type": + base = node.child_by_field_name("type") + return self._get_type_name(base) if base is not None else None + if node.type == "stable_identifier": + idents = [c for c in node.children if c.type == "identifier"] + return idents[-1].text.decode() if idents else None + return None + + def _extract_extends_types(self, def_node) -> list[str] | None: + extends_clause = def_node.child_by_field_name("extend") + if extends_clause is None: + return None + types = [] + for child in extends_clause.children: + if child.type in ("type_identifier", "generic_type", "stable_identifier"): + name = self._get_type_name(child) + if name: + types.append(name) + return types or None + + def _emit_extends_edges(self, def_node): + owner_key = self._owner_key_for_definition(def_node) + if not owner_key: + return + types = self._extract_extends_types(def_node) or [] + if not types: + return + caller_id = self._get_component_id(owner_key) + call_line = def_node.start_point[0] + 1 + for type_name in types: + if self._is_primitive(type_name): + continue + self._add_type_relationship(caller_id, type_name, call_line) + + def _emit_class_parameter_edge(self, param_node): + class_params = param_node.parent + def_node = class_params.parent if class_params is not None else None + # Scala 3 traits can take value parameters too (`trait Foo(x: T)`), + # sharing the same `class_parameters` shape as classes — restricting + # this to class_definition would silently drop their edges, even + # though _find_variable_type already resolves calls through them. + if def_node is None or def_node.type not in _TYPE_DEFINITION_NODES: + return + owner_key = self._owner_key_for_definition(def_node) + if not owner_key: + return + type_node = param_node.child_by_field_name("type") + type_name = self._get_type_name(type_node) + if type_name and not self._is_primitive(type_name): + caller_id = self._get_component_id(owner_key) + self._add_type_relationship(caller_id, type_name, param_node.start_point[0] + 1) + + def _emit_field_type_edge(self, val_node): + owner = self._find_containing_type(val_node) + if not owner: + return + type_node = val_node.child_by_field_name("type") + type_name = self._get_type_name(type_node) + if type_name and not self._is_primitive(type_name): + caller_id = self._get_component_id(owner) + self._add_type_relationship(caller_id, type_name, val_node.start_point[0] + 1) + + def _emit_instantiation_edge(self, inst_node): + caller_id = self._resolve_caller(inst_node) + if not caller_id: + return + type_node = None + for child in inst_node.children: + if child.type in ("type_identifier", "generic_type", "stable_identifier"): + type_node = child + break + type_name = self._get_type_name(type_node) + if type_name and not self._is_primitive(type_name): + self._add_type_relationship(caller_id, type_name, inst_node.start_point[0] + 1) + + def _emit_call_edge(self, call_node): + caller_id = self._resolve_caller(call_node) + if not caller_id: + return + func_node = call_node.child_by_field_name("function") + if func_node is None: + return + call_line = call_node.start_point[0] + 1 + + if func_node.type == "identifier": + self._emit_bare_call_edge(caller_id, func_node.text.decode(), call_node, call_line) + elif func_node.type == "field_expression": + self._emit_field_call_edge(caller_id, func_node, call_node, call_line) + + def _emit_bare_call_edge(self, caller_id: str, callee_name: str, call_node, call_line: int): + owner = self._find_containing_type(call_node) + candidates = [] + if owner: + candidates.append(f"{owner}.{callee_name}") + candidates.append(callee_name) + if callee_name[:1].isupper(): + # Capitalized bare calls are instantiation-shaped: a case-class + # apply or a companion object's apply. + candidates.append(f"{callee_name}$") + + resolved_id = self._resolve_candidates(candidates) + if resolved_id: + self._add_relationship_raw(caller_id, resolved_id, call_line, True) + return + if callee_name[:1].isupper() or not self._is_call_noise(callee_name): + self._add_relationship_raw(caller_id, callee_name, call_line, False) + + def _emit_field_call_edge(self, caller_id: str, func_node, call_node, call_line: int): + receiver = func_node.child_by_field_name("value") + method_node = func_node.child_by_field_name("field") + if receiver is None or method_node is None: + return + method_name = method_node.text.decode() + + if receiver.type != "identifier": + # Composite receiver (a call chain, literal, ...): keep only the + # bare method name, and only when it isn't stdlib noise. + if not self._is_call_noise(method_name): + self._add_relationship_raw(caller_id, method_name, call_line, False) + return + + receiver_name = receiver.text.decode() + + if receiver_name == "this": + owner = self._find_containing_type(call_node) + logical = f"{owner}.{method_name}" if owner else None + if logical and logical in self.top_level_nodes: + self._add_relationship_raw( + caller_id, self.top_level_nodes[logical].id, call_line, True + ) + return + if not self._is_call_noise(method_name): + self._add_relationship_raw(caller_id, method_name, call_line, False) + return + + if receiver_name[:1].isupper(): + if receiver_name in SCALA_PRIMITIVE_TYPES: + return + resolved_id = self._resolve_candidates( + [f"{receiver_name}$.{method_name}", f"{receiver_name}.{method_name}"] + ) + if resolved_id: + self._add_relationship_raw(caller_id, resolved_id, call_line, True) + return + if not self._is_call_noise(method_name): + self._add_relationship_raw( + caller_id, f"{receiver_name}.{method_name}", call_line, False + ) + return + + # Lowercase receiver: try to resolve the variable's declared type. + var_type = self._find_variable_type(call_node, receiver_name) + if var_type: + resolved_id = self._resolve_candidates( + [f"{var_type}.{method_name}", f"{var_type}$.{method_name}"] + ) + if resolved_id: + self._add_relationship_raw(caller_id, resolved_id, call_line, True) + return + if not self._is_call_noise(method_name): + self._add_relationship_raw(caller_id, f"{var_type}.{method_name}", call_line, False) + return + + if not self._is_call_noise(method_name): + self._add_relationship_raw(caller_id, method_name, call_line, False) + + def _find_variable_type(self, node, variable_name: str) -> str | None: + """Best-effort resolution of a local variable's declared type: the + enclosing method's parameters, a preceding local `val`/`var` in the + same block, the enclosing class's constructor parameters, or one of + its fields.""" + func_node = node.parent + while func_node is not None and func_node.type not in _METHOD_DEFINITION_NODES: + func_node = func_node.parent + + if func_node is not None: + # A curried method has one `parameters` node per group; check them all. + for params_node in func_node.children_by_field_name("parameters"): + for param in params_node.children: + if ( + param.type == "parameter" + and self._field_text(param, "name") == variable_name + ): + t = self._get_type_name(param.child_by_field_name("type")) + if t: + return t + + body = func_node.child_by_field_name("body") + if body is not None and body.type == "block": + for child in body.children: + if child.start_byte >= node.start_byte: + break + if ( + child.type + in ( + "val_definition", + "var_definition", + ) + and self._field_text(child, "pattern") == variable_name + ): + t = self._get_type_name(child.child_by_field_name("type")) + if t: + return t + value_node = child.child_by_field_name("value") + if value_node is not None and value_node.type == "instance_expression": + for c in value_node.children: + if c.type in ("type_identifier", "generic_type"): + return self._get_type_name(c) + + class_node = node.parent + while class_node is not None and class_node.type not in _TYPE_DEFINITION_NODES: + class_node = class_node.parent + + if class_node is not None: + class_params = class_node.child_by_field_name("class_parameters") + if class_params is not None: + for param in class_params.children: + if ( + param.type == "class_parameter" + and self._field_text(param, "name") == variable_name + ): + t = self._get_type_name(param.child_by_field_name("type")) + if t: + return t + + body = class_node.child_by_field_name("body") + if body is not None: + for child in body.children: + if ( + child.type + in ( + "val_definition", + "var_definition", + ) + and self._field_text(child, "pattern") == variable_name + ): + t = self._get_type_name(child.child_by_field_name("type")) + if t: + return t + + return None + + # ------------------------------------------------------------------ + # Resolution / noise filtering + # ------------------------------------------------------------------ + + def _resolve_candidates(self, candidates: list[str]) -> str | None: + for candidate in candidates: + node_obj = self.top_level_nodes.get(candidate) + if node_obj is not None: + return node_obj.id + return None + + def _add_type_relationship(self, caller_id: str, type_name: str, call_line: int): + resolved_id = self._resolve_candidates([type_name, f"{type_name}$"]) + if resolved_id: + self._add_relationship_raw(caller_id, resolved_id, call_line, True) + else: + self._add_relationship_raw(caller_id, type_name, call_line, False) + + def _add_relationship_raw( + self, caller: str, callee: str, call_line: int | None, resolved: bool + ): + key = (caller, callee, call_line) + if caller == callee or key in self.seen_relationships: + return + self.seen_relationships.add(key) + self.call_relationships.append( + CallRelationship( + caller=caller, callee=callee, call_line=call_line, is_resolved=resolved + ) + ) + + def _is_primitive(self, type_name: str) -> bool: + return type_name in SCALA_PRIMITIVE_TYPES + + def _is_call_noise(self, name: str) -> bool: + return name in SCALA_CORE_CALLS + + +def analyze_scala_file( + file_path: str, content: str, repo_path: str | None = None +) -> tuple[list[Node], list[CallRelationship]]: + analyzer = TreeSitterScalaAnalyzer(file_path, content, repo_path) + return analyzer.nodes, analyzer.call_relationships diff --git a/codewiki/src/be/dependency_analyzer/ast_parser.py b/codewiki/src/be/dependency_analyzer/ast_parser.py index 4554acb9..eaf93f86 100644 --- a/codewiki/src/be/dependency_analyzer/ast_parser.py +++ b/codewiki/src/be/dependency_analyzer/ast_parser.py @@ -186,6 +186,8 @@ def _file_to_module_path(self, file_path: str) -> str: ".kt", ".kts", ".rb", + ".scala", + ".sc", ] for ext in extensions: if path.endswith(ext): diff --git a/codewiki/src/be/dependency_analyzer/utils/patterns.py b/codewiki/src/be/dependency_analyzer/utils/patterns.py index c7c3f0b0..ed91a5fc 100644 --- a/codewiki/src/be/dependency_analyzer/utils/patterns.py +++ b/codewiki/src/be/dependency_analyzer/utils/patterns.py @@ -161,6 +161,7 @@ "*.kt", "*.kts", "*.scala", + "*.sc", "*.clj", "*.hs", "*.ml", @@ -287,6 +288,7 @@ ".swift": "swift", ".kt": "kotlin", ".scala": "scala", + ".sc": "scala", ".cs": "csharp", } @@ -515,6 +517,14 @@ "protected fun {name}", ], "ruby": ["def {name}", "def self.{name}"], + "scala": [ + "def {name}", + "private def {name}", + "protected def {name}", + "class {name}", + "trait {name}", + "object {name}", + ], "general": ["{name}("], # Fallback pattern } diff --git a/codewiki/src/be/prompt_template.py b/codewiki/src/be/prompt_template.py index d28944ae..37c39693 100644 --- a/codewiki/src/be/prompt_template.py +++ b/codewiki/src/be/prompt_template.py @@ -350,6 +350,8 @@ ".cs": "csharp", ".kt": "kotlin", ".kts": "kotlin", + ".scala": "scala", + ".sc": "scala", ".php": "php", ".phtml": "php", ".inc": "php", diff --git a/openspec/changes/add-scala-language-support/.openspec.yaml b/openspec/changes/add-scala-language-support/.openspec.yaml deleted file mode 100644 index e8cda9e5..00000000 --- a/openspec/changes/add-scala-language-support/.openspec.yaml +++ /dev/null @@ -1,2 +0,0 @@ -schema: spec-driven -created: 2026-09-10 diff --git a/openspec/changes/add-scala-language-support/design.md b/openspec/changes/add-scala-language-support/design.md deleted file mode 100644 index b24b360e..00000000 --- a/openspec/changes/add-scala-language-support/design.md +++ /dev/null @@ -1,233 +0,0 @@ -## Context - -See proposal.md — Why. Two aspects of the current state shape this design. - -**Scala is already half-wired.** `patterns.py` lists `*.scala` among code-file globs and -maps `".scala": "scala"` in `CODE_EXTENSIONS`. Scala files are therefore already -discovered and tagged with a language, then dropped at two later points: - -``` - [CODE_EXTENSIONS] .scala -> "scala" already wired - | - [call_graph_analyzer._find_code_files] passes through - | - [analysis_service._filter_supported_languages] DROPPED (whitelist) - | - [call_graph_analyzer._analyze_file if/elif chain] SILENT NO-OP (no branch, - | else-warning commented out) - [analyzers/scala.py] does not exist -``` - -The whitelist is the hard gate. Until `"scala"` joins it, nothing else in this change -executes, and because the dispatch fallthrough logs nothing, the failure mode during -development is silence rather than an error. - -**Leaf-node selection is typed, and its type set is narrow.** `leaf_selection.py` -defines `OOP_TYPES = {"class", "interface", "struct"}`. This drives two things: which -components survive `filter_leaf_nodes`, and the `n_oop` count feeding the issue-#75 -heuristic that decides whether free functions carry a codebase's architecture. Component -types outside that set — `trait` (emitted by PHP), `object` (Kotlin), `enum` and -`record` (Java), `delegate` (C#), `type_alias` (C++) — are already excluded from leaf -selection today. - -That exclusion is nearly invisible for existing languages because their codebases are -class-dominant. Scala is the first supported language where the excluded types *are* the -architecture: a typical Scala repository is mostly traits, objects, and case classes. A -faithful analyzer emitting `"trait"` and `"object"` would produce a repository whose -traits and objects are filtered out of documentation *and* whose `n_oop` count is near -zero, misfiring the heuristic into documenting loose methods instead of architectural -units. - -## Goals / Non-Goals - -**Goals:** - -- Land Scala support without changing behavior for any already-supported language. -- Keep the whole change inside the 14-file footprint in proposal.md — Impact. -- Model Scala's distinctive constructs (traits, companion objects) so the dependency - graph is correct rather than merely populated. - -**Non-Goals:** - -- Widening `OOP_TYPES`, adding duplicate-ID detection, or re-enabling the dispatch - warning. See proposal.md — Non-Goals; each is pre-existing and gets its own change. - Note that the `OOP_TYPES` gap is broader than PHP traits and Kotlin objects: C++ - `type_alias` components are dropped from leaf selection too, even though `cpp.py` - extracts them deliberately on the grounds that aliases are real API surface. Whichever - change widens the set should cover all of these together. -- Semantic analysis beyond syntactic extraction: no implicit resolution, no type - inference, no macro expansion. -- Treating `.sbt` as an analyzable source language (see Decision 5). - -## Decisions - -### 1. Use the two-level `component_type` / `node_type` split - -`component_type` carries the coarse type the pipeline gates on; `node_type` and -`display_name` carry the faithful Scala construct: - -| Scala construct | `component_type` | `node_type` | `display_name` | -| --- | --- | --- | --- | -| `class` / `case class` | `class` | `class` | `class Foo` | -| `trait` | `interface` | `trait` | `trait Foo` | -| `object` / `package object` | `class` | `object` | `object Foo` | -| Scala 3 `enum` | `class` | `enum` | `enum Foo` | -| method | `method` | `method` | `method Foo.bar` | -| top-level definition | `function` | `function` | `function bar` | - -*Why:* the coarse type keeps traits and objects leaf-eligible and counted toward -`n_oop`, so the issue-#75 heuristic reads a Scala repository accurately — while -`node_type` preserves what the construct actually is. No shared code changes, and no -information is discarded. - -This is an established pattern, not a new one. `typescript.py:561-563` already splits -the two, deriving `component_type` from a coarse `type` and `node_type` from a finer -`include_functions` heuristic on existing repositories. That deserves a dedicated change -`component_type`. The `Node` model carries all three fields plus `get_display_name()`. - -Critically, nothing recomputes `component_type` from `node_type`, so a deliberate split -cannot be clobbered downstream: `ast_parser._determine_component_type` — which has its -own competing whitelist including `enum`, `record`, `annotation`, and `delegate` — is -dead code, defined at `ast_parser.py:148` and never called. - -*Alternative considered — remap onto the existing vocabulary and discard the construct* -(emit `component_type="interface"` for a trait with no faithful `node_type`). Rejected -once the split was found: it loses information for no benefit, since setting the extra -fields is free. - -*Alternative considered — emit faithful `"trait"`/`"object"` as `component_type` and -widen `OOP_TYPES`.* Semantically cleanest, and it would fix the latent PHP, Kotlin, and -C++ gaps as a side effect. Rejected for this change: widening a globally shared set -changes documentation output for several shipped languages and can flip the -generated today reads from the coarse type, which the deferred `display_name` wiring -with a regression audit, not a rider on a new-language change. - -*Alternative considered — emit faithful types as `component_type` and leave `OOP_TYPES` -alone.* Rejected: it would ship Scala support that produces near-empty documentation for -idiomatic Scala. - -*Scope caveat — `node_type` and `display_name` are currently inert.* Neither flows into -generated documentation today: `get_display_name()` is never called, the MCP component -export at `mcp/tools/analysis.py:375` emits `component_type`, and `node_type` is read -only for the `artifact_file` check at `prompt_template.py:429` and a `"method"` test in -`call_graph_analyzer.py`. So setting these fields preserves the construct in the graph -and the exported artifacts at no cost, but it does **not** by itself make the docs say -"trait" rather than "interface". Surfacing faithful labels in documentation means wiring -`display_name` through the prompt path — a small change, deliberately not in this scope. -This design leaves the seam clean for it, and for the deferred `OOP_TYPES` work. - -### 2. Suffix companion objects in the component identifier - -Component IDs follow `relpath::Name` / `relpath::Class.method`. In Scala a companion -object is a *top-level peer* sharing its class's name, so `class Buffer` and -`object Buffer` both claim `buffer.scala::Buffer`. Because components are stored by -plain dict assignment with no collision check, the later-parsed declaration silently -displaces the earlier one — the class disappears from the graph while its methods remain -parented to the surviving object's node. - -The object therefore takes a suffixed identifier: `buffer.scala::Buffer, mirroring -Scala's own JVM encoding of module classes. Members follow their owner -(`buffer.scala::Buffer.push`, `buffer.scala::Buffer$.apply`). - -*Why:* both declarations survive as distinct components with their own source ranges and -documentation, and every edge stays attributable to the declaration that produced it. -The ` convention is one Scala developers already recognize from stack traces. - -*Alternative considered — merge the companion's members into the class as static -members.* Closer to how developers think about companions, but it conflates two -disjoint source ranges into one component, so the generated documentation would show one -declaration's source text under a component holding the other's members. - -*Note:* this is a local encoding choice, not a fix for the general silent-overwrite -defect, which is deferred. - -### 3. Use the dedicated `tree-sitter-scala` package - -*Why:* every one of the ten existing analyzers imports its own `tree_sitter_` -module and constructs a `Parser` from it. Following that pattern keeps the new analyzer -reviewable against its siblings. - -*Alternative considered — `tree_sitter_language_pack`,* already a dependency and already -capable of Scala. Rejected as inconsistent with the established per-language pattern; -switching parser sourcing is a repo-wide decision, not a Scala one. - -Version 0.26.2 introduces no `tree-sitter` core constraint in practice: its only core -pin is the optional `core` extra (`tree-sitter~=0.22`), which the pinned 0.23.2 already -satisfies. Wheels cover every CI platform. The grammar covers Scala 2 and 3, confirmed -against its published node types, which include `trait_definition`, -`object_definition`, `package_object`, `enum_definition`, `given_definition`, and -`extension_definition`. - -### 4. Register the language at every gate, not just the analyzer - -The whitelists in `analysis_service.py` (both the filter and the reported -supported-language list), the CLI's `SUPPORTED_EXTENSIONS` and language-detection map, -and the MCP incremental `source_extensions` set are all independent allowlists. Each -needs `.scala`/`.sc` or `"scala"` added. The Ruby precedent touched all of them; the -original requirements write-up named only some. - -*Why it matters:* the analyzer being correct is not sufficient for the feature to work. -Missing `repo_validator.py` alone makes the CLI reject a pure-Scala repository outright. - -### 5. Treat `.sbt` as a build artifact, never as analyzable source - -`build.sbt` and `project/*.sbt` join the manifest and build classifications in -`artifact.py` alongside the existing `pom.xml`, `build.gradle`, and `build.gradle.kts` -entries. `.sbt` is *not* added to `CODE_EXTENSIONS`. - -*Why:* `.sbt` files are syntactically Scala but semantically build configuration. -Documenting them as application source would misrepresent the architecture, while -leaving them unclassified means an sbt project's build story goes undocumented — a -visible gap now that artifact-aware generation has landed. - -`project/*.scala` (sbt meta-build code) remains ordinary Scala source. Special-casing it -adds a path-shaped exception for modest benefit. - -## Risks / Trade-offs - -**The Kotlin analyzer is a structural template, not a copy-paste source.** Kotlin's -grammar uses `class_declaration` / `object_declaration` / `function_declaration`; Scala's -uses `class_definition` / `object_definition` / `function_definition`. Every `node.type` -comparison needs rewriting. → Treat Kotlin as the reference for *shape* (node -extraction, then relationship extraction, module-path derivation, ID construction) and -derive all node type names from the Scala grammar's own node types. - -**Scala 3 significant-indentation syntax may parse poorly.** This is the only unknown -that could invalidate the extraction approach. → Spike first: parse representative -Scala 2 and Scala 3 files and assert the tree contains no `ERROR` nodes before building -extraction on top. Sequenced as the first task. - -**Scala's expression-oriented style may produce noisy call graphs.** Heavy chaining, -higher-order functions, and for-comprehensions generate many call expressions against -standard-library targets. → Follow the precedent of the existing analyzers and filter -primitives and common built-ins, as Kotlin and PHP already do with their primitive sets; -assert the exclusion in tests. - -**`.sc` is not exclusive to Scala** — SuperCollider and Scilab also use it. → Accepted: -the file is parsed with the Scala grammar and a non-Scala file simply yields no -components, matching how the other analyzers behave on unparseable input. - -**Decision 1's coarse type is still a compromise.** A Scala trait carries -`component_type="interface"`, so any consumer reading only the coarse type sees an -interface. → Largely mitigated by the split: `node_type="trait"` and -`display_name="trait Foo"` keep the construct in the graph, so no information is lost -and a future consumer can render it faithfully. The residual issue is that documentation -would address. -`ruby.py:608-611` sets `node_type` and a `" "` `display_name` alongside -`subtype` (a TS type alias is `component_type="type"`, `node_type="type_alias"`). - -## Open Questions - -These can be answered during implementation without changing the specs, the approach, -or the task breakdown: - -- **Scala 3 `given` definitions.** Typeclass instances are architecturally meaningful, - but whether each is a *documentable component* is unclear. Leaning toward extracting - named givens as `"class"` and skipping anonymous ones. Requires a real Scala 3 - codebase to judge signal versus noise. -- **Scala 3 `extension` blocks.** The methods inside are plausibly `"method"` components, - but their owner is the extended type, which may live outside the repository. -- **`type` aliases.** The existing `"type_alias"` component type sits outside - `OOP_TYPES` and would be dropped; likely skip rather than mismap. -- **`project/build.properties`.** Whether the sbt version pin is worth classifying as a - build artifact, or is too granular to document. diff --git a/openspec/changes/add-scala-language-support/proposal.md b/openspec/changes/add-scala-language-support/proposal.md deleted file mode 100644 index e5627a41..00000000 --- a/openspec/changes/add-scala-language-support/proposal.md +++ /dev/null @@ -1,87 +0,0 @@ -## Why - -CodeWiki supports ten languages but not Scala, and users have asked for it to unify -JVM workflows (issue #105). The groundwork is already partly in place — `.scala` is -registered in `CODE_EXTENSIONS` and the tree-sitter Scala grammar exists — but Scala -files are recognized, tagged `language: "scala"`, and then silently discarded before -any analyzer runs. Today a Scala repository produces no documentation and no error. - -## What Changes - -- Add a tree-sitter Scala analyzer (`analyzers/scala.py`) that extracts classes, - traits, objects, enums, and methods and emits call/inheritance relationships. -- Register `.scala` and `.sc` as analyzable source across discovery, dispatch, - validation, and incremental-change detection. -- Admit `"scala"` to the two language whitelists in `analysis_service.py`. This is the - hard gate: without it every other change in this proposal is unreachable code. -- Classify `build.sbt` and `project/*.sbt` as build manifests so sbt-based projects get - their build story documented alongside the existing Maven/Gradle handling. -- Add `tree-sitter-scala` to `pyproject.toml` and `requirements.txt`. -- Add `tests/test_scala_analyzer.py` covering extraction, relationships, and an - end-to-end `DependencyParser` pass. -- Update the supported-languages list in `README.md`. - -No breaking changes. Existing languages are untouched: Scala constructs are mapped onto -the current `component_type` vocabulary rather than widening the shared `OOP_TYPES` set -(see design.md). - -## Capabilities - -### New Capabilities -- `scala-language-support`: Discovery, parsing, component extraction, and dependency - analysis for Scala source files, plus classification of sbt build files. - -### Modified Capabilities - -None. `openspec/specs/` is currently empty, so no existing capability's requirements -change. - -## Impact - -**Dependencies.** Adds `tree-sitter-scala` (0.26.2). No `tree-sitter` core bump: its -only core pin is the optional `core` extra (`tree-sitter~=0.22`), which the pinned -0.23.2 already satisfies. abi3-cp39 wheels are published for macOS x86_64/arm64, -manylinux x86_64/aarch64, musllinux, and Windows amd64/arm64, so no source builds in -CI. The grammar covers both Scala 2 and Scala 3. - -**Affected code** — 14 files, matching the footprint of the Ruby precedent (PR #97, -commit `05c7576`): - -| File | Change | -| --- | --- | -| `codewiki/src/be/dependency_analyzer/analyzers/scala.py` | new analyzer | -| `codewiki/src/be/dependency_analyzer/analysis/analysis_service.py` | `"scala"` in both language whitelists | -| `codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py` | dispatch branch, `lang-scala` node class | -| `codewiki/src/be/dependency_analyzer/utils/patterns.py` | `.sc` mapping, Scala function patterns | -| `codewiki/src/be/dependency_analyzer/ast_parser.py` | strip `.scala`/`.sc` in module paths | -| `codewiki/src/be/dependency_analyzer/analyzers/artifact.py` | source exts, sbt manifests | -| `codewiki/src/be/prompt_template.py` | `.scala`/`.sc` fence-language mapping | -| `codewiki/cli/utils/repo_validator.py` | `SUPPORTED_EXTENSIONS` | -| `codewiki/cli/utils/validation.py` | `detect_languages` map | -| `codewiki/mcp/tools/analysis.py` | `source_extensions` for incremental detection | -| `pyproject.toml`, `requirements.txt` | dependency | -| `tests/test_scala_analyzer.py` | new tests | -| `README.md` | supported-languages list | - -The last six are absent from the original requirements write-up but were all touched -when Ruby was added. Three of them are user-visible: without `repo_validator.py` the -CLI rejects a pure-Scala repository as containing no supported code, without -`validation.py` Scala is missing from detected-language statistics, and without -`mcp/tools/analysis.py` edits to `.scala` files never trigger incremental -regeneration. - -## Non-Goals - -Three adjacent defects were found while scoping this change. All are pre-existing, none -are Scala-specific, and each is deferred to its own change: - -- **`OOP_TYPES` excludes `trait`, `object`, `enum`, and `record`.** PHP already emits - `"trait"` and Kotlin `"object"`, so both are already dropped from leaf-node selection. - Widening the set is the semantically correct fix but changes documentation output for - three shipped languages and needs a regression audit. -- **Duplicate component IDs overwrite silently.** `ast_parser.py:109` and the ten - `self.functions[func_id] = func` sites in `call_graph_analyzer.py` are plain dict - assignments with no collision detection. -- **Unsupported-language dispatch is silent.** The `else: logger.warning(...)` at - `call_graph_analyzer.py:257` is commented out, which is why Scala files currently - fail without a diagnostic. diff --git a/openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md b/openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md deleted file mode 100644 index 611f64a1..00000000 --- a/openspec/changes/add-scala-language-support/specs/scala-language-support/spec.md +++ /dev/null @@ -1,143 +0,0 @@ -## Purpose - -Lets CodeWiki analyze and document Scala repositories on the same terms as the other -supported languages: recognizing Scala source files, extracting their architectural -components and dependency relationships, and classifying sbt build files. - -## ADDED Requirements - -### Requirement: Scala source file recognition - -The system SHALL treat files with the `.scala` and `.sc` extensions as analyzable -source code, for both Scala 2 and Scala 3 syntax. - -#### Scenario: Repository containing only Scala sources is accepted - -- **WHEN** a user requests documentation for a repository whose only source files are `.scala` files -- **THEN** the repository passes validation instead of being rejected for containing no supported code - -#### Scenario: Scala appears in detected language statistics - -- **WHEN** a repository containing Scala files is scanned for language composition -- **THEN** Scala is reported with a file count reflecting its `.scala` and `.sc` files - -#### Scenario: Scala 3 syntax is parsed - -- **WHEN** a source file uses Scala 3 constructs such as `enum`, `given`, `extension`, or significant-indentation blocks -- **THEN** the file is parsed without error and its top-level definitions are extracted - -### Requirement: Scala component extraction - -The system SHALL extract Scala classes, case classes, traits, objects, package objects, -enums, and their members as documentable components, each carrying its name, source -location, source text, parameters, and documentation comment when present. - -#### Scenario: Traits and objects are documentable architectural units - -- **WHEN** a Scala file declares a trait and a standalone object -- **THEN** both are extracted as components eligible for documentation, not discarded - -#### Scenario: Methods are attributed to their enclosing type - -- **WHEN** a method is declared inside a class, trait, or object -- **THEN** the resulting component identifies that enclosing type as its owner - -#### Scenario: Top-level definitions are extracted as functions - -- **WHEN** a Scala file declares a definition outside any class, trait, or object -- **THEN** it is extracted as a free function component - -#### Scenario: Documentation comments are captured - -- **WHEN** a Scala declaration is preceded by a Scaladoc or line comment -- **THEN** the component records that text as its documentation - -### Requirement: Companion declarations remain distinct - -A class or trait and its same-named companion object SHALL both be preserved as -separate components with distinct identifiers. Neither declaration may displace the -other. - -#### Scenario: Class and companion object both survive - -- **WHEN** a file declares both `class Buffer` and `object Buffer` -- **THEN** two distinct components exist, each retaining its own source range, documentation, and members - -#### Scenario: Members are attributed to the correct companion - -- **WHEN** `class Buffer` declares `push` and `object Buffer` declares `apply` -- **THEN** `push` is attributed to the class and `apply` to the object, not both to one of them - -### Requirement: Scala dependency and call relationships - -The system SHALL emit dependency relationships for Scala code covering inheritance, -trait mixins, constructor parameter and field types, instantiation, and method calls. -Relationships whose target is defined in the analyzed repository SHALL resolve to that -component; relationships to symbols outside the repository SHALL be retained as -unresolved logical names. - -#### Scenario: Inheritance produces a dependency edge - -- **WHEN** a class extends a base class defined in another file in the repository -- **THEN** a resolved dependency edge links the subclass to that base class - -#### Scenario: Trait mixins produce dependency edges - -- **WHEN** a class mixes in a trait defined in the repository -- **THEN** a resolved dependency edge links the class to that trait - -#### Scenario: Intra-file calls resolve to sibling components - -- **WHEN** a method calls another method on the same type -- **THEN** a resolved dependency edge links the two method components - -#### Scenario: External symbols stay unresolved - -- **WHEN** Scala code references a type or method from the standard library or a third-party dependency -- **THEN** the relationship is recorded as an unresolved logical name rather than a repository component - -#### Scenario: Standard-library noise is excluded - -- **WHEN** Scala code uses ubiquitous built-ins such as primitive types or common collection operations -- **THEN** those references do not appear as dependency edges - -### Requirement: sbt build files are classified as build artifacts - -The system SHALL classify `build.sbt` and `.sbt` files under a `project/` directory as -build manifests, so that sbt-based projects have their build and packaging story -documented alongside the existing Maven and Gradle handling. - -#### Scenario: build.sbt is treated as a manifest - -- **WHEN** a repository contains a `build.sbt` at its root -- **THEN** that file is classified as a build manifest artifact rather than ignored or documented as application source - -#### Scenario: sbt plugin definitions are treated as build files - -- **WHEN** a repository contains `project/plugins.sbt` -- **THEN** that file is classified as a build artifact - -### Requirement: End-to-end documentation generation for Scala repositories - -A Scala repository SHALL produce documentation describing its actual components. Scala -files may not be silently dropped between discovery and analysis. - -#### Scenario: Scala components reach the generated documentation - -- **WHEN** documentation is generated for a repository of Scala source files -- **THEN** the generated output describes components extracted from those files - -#### Scenario: Scala code snippets are tagged for highlighting - -- **WHEN** generated documentation embeds a code snippet taken from a Scala source file -- **THEN** the snippet is tagged as Scala so it renders with Scala syntax highlighting - -### Requirement: Incremental regeneration reacts to Scala edits - -The system SHALL detect modifications to Scala source files when determining whether a -repository has changed since its last documentation run. - -#### Scenario: A modified Scala file marks the repository as changed - -- **WHEN** a `.scala` file is modified after the previous documentation run -- **THEN** the change detection reports that file as changed diff --git a/openspec/changes/add-scala-language-support/tasks.md b/openspec/changes/add-scala-language-support/tasks.md deleted file mode 100644 index a1572034..00000000 --- a/openspec/changes/add-scala-language-support/tasks.md +++ /dev/null @@ -1,56 +0,0 @@ -## 1. Grammar Spike - -- [ ] 1.1 Add `tree-sitter-scala>=0.26.2` to `pyproject.toml` and pin `tree-sitter-scala==0.26.2` in `requirements.txt`; verify `pip install -e .` succeeds and `python -c "import tree_sitter_scala"` imports without a source build -- [ ] 1.2 Parse a representative Scala 2 file and a Scala 3 file using significant-indentation syntax, `enum`, `given`, and `extension`; verify neither tree contains `ERROR` or `MISSING` nodes. If Scala 3 indentation fails, stop and revisit the design before continuing — this gates the extraction approach -- [ ] 1.3 Record the concrete grammar node type names for class, case class, trait, object, package object, enum, method, and top-level definitions; verify each name appears in the grammar's node types rather than being carried over from the Kotlin analyzer - -## 2. Analyzer Core - -- [ ] 2.1 Create `codewiki/src/be/dependency_analyzer/analyzers/scala.py` with a `TreeSitterScalaAnalyzer` class and an `analyze_scala_file(file_path, content, repo_path)` entry point mirroring the Kotlin analyzer's structure; verify it returns empty node and relationship lists for an empty file without raising -- [ ] 2.2 Implement module-path and relative-path derivation stripping `.scala` and `.sc`; verify a nested file yields a dotted module path with the extension removed -- [ ] 2.3 Implement component ID construction, giving companion objects the `suffix per design Decision 2; verify a file declaring both` class Buffer `and` object Buffer `yields` buffer.scala::Buffer `and` buffer.scala::Buffer as distinct components with neither displaced -- [ ] 2.4 Extract classes, case classes, traits, objects, package objects, and Scala 3 enums, applying the type mapping from design Decision 1; verify a trait is emitted with component type `interface` and an object with `class` -- [ ] 2.5 Extract methods with their enclosing type as owner, and top-level definitions as free functions; verify a method inside a trait is attributed to that trait and a top-level `def` is emitted as a function -- [ ] 2.6 Extract parameters, source ranges, and Scaladoc or line-comment documentation for each component; verify a documented method reports its parameter list and a non-empty docstring - -## 3. Relationship Extraction - -- [ ] 3.1 Emit inheritance and trait-mixin edges from `extends`/`with` clauses; verify a class extending a repository-local base class and mixing in a repository-local trait produces a resolved edge for each -- [ ] 3.2 Emit edges for constructor parameter types, field types, and instantiation; verify a class with a field of a repository-local type produces a resolved edge to it -- [ ] 3.3 Emit method-call edges, resolving intra-file targets to sibling components and leaving external targets as unresolved logical names; verify an intra-type call resolves and a standard-library call does not -- [ ] 3.4 Define a Scala primitive and common built-in exclusion set following the Kotlin and PHP precedent; verify references to primitive types and ubiquitous collection operations produce no edges - -## 4. Pipeline Registration - -- [ ] 4.1 Add `"scala"` to both language sets in `analysis/analysis_service.py` (`_filter_supported_languages` and `_get_supported_languages`); verify Scala files are no longer dropped before dispatch. This is the hard gate — without it tasks 2 and 3 are unreachable -- [ ] 4.2 Add the `scala` dispatch branch and an `_analyze_scala_file` method to `analysis/call_graph_analyzer.py`, plus the `lang-scala` node class alongside the existing `lang-kotlin` handling; verify a Scala file in a test repository produces components end to end -- [ ] 4.3 Map `.sc` to `"scala"` in `utils/patterns.py` `CODE_EXTENSIONS`, add `*.sc` to the code-file globs, and add Scala entries to `FUNCTION_DEFINITION_PATTERNS`; verify `.scala` and `.sc` both resolve to the `scala` language -- [ ] 4.4 Add `.scala` and `.sc` to the extension list in `ast_parser.py` `_file_to_module_path`; verify module paths for Scala files drop the extension -- [ ] 4.5 Add `.scala` and `.sc` to the source-extension set in `analyzers/artifact.py` so Scala files classify as code rather than falling through to artifact detection; verify a plain `.scala` file is not classified as an artifact -- [ ] 4.6 Add `.scala` and `.sc` to the fence-language map in `src/be/prompt_template.py`; verify a Scala snippet is tagged `scala` - -## 5. sbt Build File Classification - -- [ ] 5.1 Add `build.sbt` to `_MANIFEST_NAMES` and `.sbt` handling for `project/` files in `analyzers/artifact.py`; verify `build.sbt` classifies as a manifest and `project/plugins.sbt` as a build artifact -- [ ] 5.2 Confirm `.sbt` is absent from `CODE_EXTENSIONS` per design Decision 5; verify an `.sbt` file is never dispatched to the Scala analyzer - -## 6. CLI and MCP Gates - -- [ ] 6.1 Add `.scala` and `.sc` to `SUPPORTED_EXTENSIONS` in `cli/utils/repo_validator.py`; verify a repository containing only Scala files passes validation instead of being rejected as having no supported code -- [ ] 6.2 Add a `"Scala": [".scala", ".sc"]` entry to `detect_languages` in `cli/utils/validation.py`; verify Scala appears with a file count in detected language statistics -- [ ] 6.3 Add `.scala` and `.sc` to `source_extensions` in `mcp/tools/analysis.py`; verify a modified `.scala` file is reported as changed by incremental detection - -## 7. Tests - -- [ ] 7.1 Create `tests/test_scala_analyzer.py` modeled on `tests/test_ruby_analyzer.py`, with `pytest.importorskip("tree_sitter_scala")` and a Scala sample exercising a trait, a class with a companion object, inheritance, a mixin, an intra-type call, and a top-level definition; verify the suite runs -- [ ] 7.2 Add extraction assertions covering component types, owners, IDs, and the companion ` suffix; verify all pass -- [ ] 7.3 Add relationship assertions covering resolved inheritance, resolved mixin, resolved intra-file call, unresolved external target, and excluded built-in noise; verify all pass -- [ ] 7.4 Add a `DependencyParser` end-to-end test across two Scala files asserting cross-file inheritance resolves into `depends_on`; verify it passes -- [ ] 7.5 Add a Scala 3 sample covering `enum` and significant-indentation syntax; verify components are extracted -- [ ] 7.6 Add an artifact-classification test for `build.sbt` and `project/plugins.sbt`; verify both classify as build artifacts -- [ ] 7.7 Run the full existing test suite; verify no regressions in the Python, Kotlin, PHP, Ruby, or artifact analyzer tests - -## 8. Documentation - -- [ ] 8.1 Add Scala to the supported-languages list in `README.md`; verify it renders alongside the existing ten entries -- [ ] 8.2 Resolve the design's open questions on `given`, `extension`, `type` aliases, and `project/build.properties` against a real Scala 3 repository, and record the outcomes in `design.md`; verify each open question is either answered or explicitly carried forward diff --git a/openspec/changes/archive/.gitkeep b/openspec/changes/archive/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/openspec/config.yaml b/openspec/config.yaml deleted file mode 100644 index c4d34ace..00000000 --- a/openspec/config.yaml +++ /dev/null @@ -1,32 +0,0 @@ -schema: spec-driven - -# Project context (optional) -# This is shown to AI when creating artifacts. -# Add your tech stack, conventions, style guides, domain knowledge, etc. -# Example: -# context: | -# Tech stack: TypeScript, React, Node.js -# We use conventional commits -# Domain: e-commerce platform - -# Per-artifact rules (optional) -# Add custom rules for specific artifacts. -# Example: -# rules: -# proposal: -# - Keep proposals under 500 words -# - Always include a "Non-goals" section -# tasks: -# - Break tasks into chunks of max 2 hours - -# Per-operation guidance (optional) -# Add advisory guidance for how apply and archive work should be conducted. -# This is separate from artifact rules above. -# Example: -# operations: -# apply: -# guidance: -# - Keep test summaries concise -# archive: -# guidance: -# - Summarize the archive outcome before finishing diff --git a/openspec/specs/.gitkeep b/openspec/specs/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/pyproject.toml b/pyproject.toml index 2675acfe..50ad1d8e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,6 +40,7 @@ dependencies = [ "tree-sitter-php>=0.23.0", "tree-sitter-kotlin>=1.1.0", "tree-sitter-ruby>=0.23.1", + "tree-sitter-scala>=0.23.2,<0.24.0", "openai>=1.107.0", "litellm>=1.77.0", "pydantic>=2.11.7", diff --git a/requirements.txt b/requirements.txt index a38713ea..ef6259b0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -151,6 +151,7 @@ tree-sitter-kotlin==1.1.0 tree-sitter-language-pack==0.8.0 tree-sitter-python==0.23.6 tree-sitter-ruby==0.23.1 +tree-sitter-scala==0.23.4 tree-sitter-typescript==0.21.2 tree-sitter-yaml==0.7.1 types-protobuf==6.30.2.20250822 diff --git a/tests/test_scala_analyzer.py b/tests/test_scala_analyzer.py new file mode 100644 index 00000000..0789966e --- /dev/null +++ b/tests/test_scala_analyzer.py @@ -0,0 +1,254 @@ +"""Tests for the tree-sitter based Scala analyzer.""" + +from pathlib import Path + +import pytest + +pytest.importorskip("tree_sitter_scala") + +from codewiki.src.be.dependency_analyzer.analyzers.artifact import ( + ArtifactOptions, + classify_artifact, +) +from codewiki.src.be.dependency_analyzer.analyzers.scala import analyze_scala_file +from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser + +SAMPLE = """\ +package pipeline + +/** Base contract for buffers that can be flushed. */ +trait Flushable { + def flush(): Unit +} + +/** Structured logger used by buffers. */ +class Logger { + def warn(message: String): Unit = () +} + +/** Buffers events before flushing them downstream. */ +class Buffer(capacity: Int, logger: Logger) extends BaseBuffer with Flushable { + + def push(item: Int): Unit = { + validate(item) + logger.warn("pushed") + ExternalService.ping() + } + + def validate(item: Int): Unit = { + require(item >= 0) + } + + def flush(): Unit = { + println("flushing") + } +} + +/** Companion object for Buffer. */ +object Buffer { + def apply(capacity: Int): Buffer = new Buffer(capacity, new Logger()) +} + +def standaloneHelper(value: Int): Int = value + 1 +""" + +SCALA3_SAMPLE = """\ +package shapes + +enum Color: + case Red, Green, Blue + +trait Shape: + def area: Double + +case class Circle(radius: Double) extends Shape: + def area: Double = Math.PI * radius * radius +""" + + +def _analyze(tmp_path: Path): + file_path = tmp_path / "buffer.scala" + file_path.write_text(SAMPLE, encoding="utf-8") + return analyze_scala_file(str(file_path), SAMPLE, repo_path=str(tmp_path)) + + +def test_extracts_traits_classes_objects_and_methods(tmp_path: Path) -> None: + nodes, _ = _analyze(tmp_path) + by_name = {node.name: node for node in nodes} + + assert by_name["Flushable"].component_type == "interface" + assert by_name["Flushable"].node_type == "trait" + + assert by_name["Buffer"].component_type == "class" + assert by_name["Buffer"].node_type == "class" + assert by_name["Buffer"].base_classes == ["BaseBuffer", "Flushable"] + + assert by_name["Buffer.push"].component_type == "method" + assert by_name["Buffer.push"].class_name == "Buffer" + assert by_name["Buffer.validate"].component_type == "method" + assert by_name["standaloneHelper"].component_type == "function" + + # Companion object survives as a distinct, `$`-suffixed component. + assert by_name["Buffer$"].component_type == "class" + assert by_name["Buffer$"].node_type == "object" + assert by_name["Buffer$.apply"].component_type == "method" + assert by_name["Buffer$.apply"].class_name == "Buffer$" + + assert by_name["Buffer"].id == "buffer.scala::Buffer" + assert by_name["Buffer$"].id == "buffer.scala::Buffer$" + assert by_name["Buffer.push"].id == "buffer.scala::Buffer.push" + assert by_name["Buffer$.apply"].id == "buffer.scala::Buffer$.apply" + assert all(node.language == "scala" for node in nodes) + + +def test_extracts_docstring_and_parameters(tmp_path: Path) -> None: + nodes, _ = _analyze(tmp_path) + by_name = {node.name: node for node in nodes} + + assert by_name["Buffer"].has_docstring + assert "Buffers events" in by_name["Buffer"].docstring + assert by_name["Buffer.push"].parameters == ["item: Int"] + + +def test_extracts_call_relationships(tmp_path: Path) -> None: + _, relationships = _analyze(tmp_path) + edges = {(rel.caller, rel.callee, rel.is_resolved) for rel in relationships} + + # Inheritance: BaseBuffer lives in another file, stays an unresolved logical name. + assert ("buffer.scala::Buffer", "BaseBuffer", False) in edges + # Trait mixin resolved within the same file. + assert ("buffer.scala::Buffer", "buffer.scala::Flushable", True) in edges + + # Constructor parameter of a repository-local type. + assert ("buffer.scala::Buffer", "buffer.scala::Logger", True) in edges + + # Intra-type call resolves to the sibling method. + assert ("buffer.scala::Buffer.push", "buffer.scala::Buffer.validate", True) in edges + # Call through a constructor-parameter-typed receiver resolves. + assert ("buffer.scala::Buffer.push", "buffer.scala::Logger.warn", True) in edges + # External symbol stays an unresolved logical name. + assert ("buffer.scala::Buffer.push", "ExternalService.ping", False) in edges + + # Instantiation edges from the companion's factory method. + assert ("buffer.scala::Buffer$.apply", "buffer.scala::Buffer", True) in edges + assert ("buffer.scala::Buffer$.apply", "buffer.scala::Logger", True) in edges + + # Standard-library noise is excluded. + assert not any(rel.callee.endswith("require") for rel in relationships) + assert not any(rel.callee.endswith("println") for rel in relationships) + + +def test_scala3_enum_and_indentation_syntax(tmp_path: Path) -> None: + file_path = tmp_path / "shapes.scala" + file_path.write_text(SCALA3_SAMPLE, encoding="utf-8") + nodes, _ = analyze_scala_file(str(file_path), SCALA3_SAMPLE, repo_path=str(tmp_path)) + by_name = {node.name: node for node in nodes} + + assert by_name["Color"].component_type == "class" + assert by_name["Color"].node_type == "enum" + assert by_name["Shape"].component_type == "interface" + assert by_name["Circle"].component_type == "class" + assert by_name["Circle"].base_classes == ["Shape"] + assert by_name["Circle.area"].component_type == "method" + + +def test_empty_file_returns_no_components(tmp_path: Path) -> None: + file_path = tmp_path / "empty.scala" + file_path.write_text("", encoding="utf-8") + nodes, relationships = analyze_scala_file(str(file_path), "", repo_path=str(tmp_path)) + assert nodes == [] + assert relationships == [] + + +def test_curried_method_parameters_are_not_truncated(tmp_path: Path) -> None: + src = """\ +class Curried { + def add(x: Int)(y: Int): Int = x + y +} +""" + file_path = tmp_path / "curried.scala" + file_path.write_text(src, encoding="utf-8") + nodes, _ = analyze_scala_file(str(file_path), src, repo_path=str(tmp_path)) + by_name = {node.name: node for node in nodes} + + # A curried method has one `parameters` node per group; both must survive. + assert by_name["Curried.add"].parameters == ["x: Int", "y: Int"] + + +def test_parameterized_trait_constructor_edge_resolves(tmp_path: Path) -> None: + src = """\ +class Logger { + def warn(message: String): Unit = () +} + +trait Parameterized(logger: Logger) { + def use(): Unit = logger.warn("x") +} +""" + file_path = tmp_path / "param_trait.scala" + file_path.write_text(src, encoding="utf-8") + _, relationships = analyze_scala_file(str(file_path), src, repo_path=str(tmp_path)) + edges = {(rel.caller, rel.callee, rel.is_resolved) for rel in relationships} + + # Scala 3 traits can take value parameters via the same `class_parameters` + # shape as classes; the constructor-parameter edge must not be dropped. + assert ("param_trait.scala::Parameterized", "param_trait.scala::Logger", True) in edges + + +def test_unresolved_non_noise_calls_stay_unresolved(tmp_path: Path) -> None: + src = """\ +class Worker { + def run(param: Unknown): Unit = { + doWork() + param.process() + val x = compute() + x.finish() + } +} +""" + file_path = tmp_path / "worker.scala" + file_path.write_text(src, encoding="utf-8") + _, relationships = analyze_scala_file(str(file_path), src, repo_path=str(tmp_path)) + edges = {(rel.caller, rel.callee, rel.is_resolved) for rel in relationships} + + caller = "worker.scala::Worker.run" + # Bare call with no receiver and no in-file match: unresolved bare name. + assert (caller, "doWork", False) in edges + # Receiver type resolves (constructor param), but the method itself doesn't. + assert (caller, "Unknown.process", False) in edges + # Receiver type can't be inferred at all: falls back to the bare method name. + assert (caller, "finish", False) in edges + + +def test_sc_extension_reaches_the_scala_analyzer(tmp_path: Path) -> None: + (tmp_path / "script.sc").write_text( + 'object Hello {\n def greet(): String = "hi"\n}\n', encoding="utf-8" + ) + components = DependencyParser(str(tmp_path)).parse_repository() + assert "script.sc::Hello$" in components + assert "script.sc::Hello$.greet" in components + + +def test_sbt_files_classify_as_build_artifacts() -> None: + opts = ArtifactOptions() + assert classify_artifact("build.sbt", "build.sbt", 100, opts) == "manifest" + assert classify_artifact("project/plugins.sbt", "plugins.sbt", 50, opts) == "build" + + +def test_dependency_parser_end_to_end(tmp_path: Path) -> None: + (tmp_path / "base_buffer.scala").write_text( + "package pipeline\n\nclass BaseBuffer {\n def flush(): Unit = ()\n}\n", + encoding="utf-8", + ) + (tmp_path / "buffer.scala").write_text(SAMPLE, encoding="utf-8") + + components = DependencyParser(str(tmp_path)).parse_repository() + + assert "buffer.scala::Buffer" in components + assert "buffer.scala::Buffer.push" in components + assert "base_buffer.scala::BaseBuffer" in components + + # The cross-file inheritance edge resolves during global resolution. + assert "base_buffer.scala::BaseBuffer" in components["buffer.scala::Buffer"].depends_on + # The intra-file call edge survives into depends_on. + assert "buffer.scala::Buffer.validate" in components["buffer.scala::Buffer.push"].depends_on From 737a5e3402a1ec9d7b9777e5cd825c9080b6fe54 Mon Sep 17 00:00:00 2001 From: Mageshwaran Rajendran Date: Mon, 14 Sep 2026 13:03:06 -0400 Subject: [PATCH 3/3] Removing a local content --- explore_scala.md | 30 ------------------------------ 1 file changed, 30 deletions(-) delete mode 100644 explore_scala.md diff --git a/explore_scala.md b/explore_scala.md deleted file mode 100644 index fe98fc2a..00000000 --- a/explore_scala.md +++ /dev/null @@ -1,30 +0,0 @@ -# Explore Scala Language Support - -## Requirements Specification: Scala Language Support -**Issue Reference:** #105 -**Status:** Approved for Community Contribution -**Primary Objective:** Implement parsing and dependency analysis support for the Scala programming language within CodeWiki. - -### 1. Context & Background -CodeWiki currently supports Java, JavaScript, Python, Ruby, and Kotlin. Users have requested Scala support to unify their workflows. Because the core engine utilizes AST Tree-sitter—which natively supports Scala—the foundational groundwork for this integration already exists. Due to its JVM-based nature, the Scala implementation will closely mirror the existing Java and Kotlin integrations. -### 2. Functional Requirements -- The system must successfully identify and parse files with the `.scala` extension. -- The system must be able to generate an Abstract Syntax Tree (AST) for Scala codebases. -- The system must correctly analyze dependencies and call graphs for Scala services. -### 3. Technical Implementation Requirements -The implementation requires updates across dependency management, core analyzer logic, and extension routing. -**3.1 Dependency Management** -- Update `pyproject.toml` to include the `tree-sitter-scala` package. -**3.2 Core Analyzer Creation** -- **File:** Create a new analyzer module at `codewiki/src/be/dependency_analyzer/analyzers/scala.py`. -- **Logic:** The analyzer should handle the JVM/package model specific to Scala. -- **Reference:** Use the existing Kotlin and Java analyzers as architectural templates. -**3.3 File Extension Registration** The `.scala` file extension must be registered in the existing routing and parsing modules. Update the following files to include `.scala` handling (similar to existing `.kt` configurations): -- `utils/patterns.py` -- `ast_parser.py` -- `analysis/call_graph_analyzer.py` -- `analyzers/artifact.py` -- `prompt_template.py` -### 4. Testing Requirements -- **Unit Tests:** Create a dedicated test file (e.g., `tests/test_scala_analyzer.py`) modeled after `tests/test_ruby_analyzer.py`. -- **Test Data:** Include at least one sample Scala code snippet to validate that the AST parsing and dependency analysis function correctly end-to-end. \ No newline at end of file