Skip to content

Import: share assembly CCUs between projects - #20296

Merged
T-Gro merged 12 commits into
dotnet:mainfrom
auduchinok:il-shareImportedAssemblies
Aug 27, 2026
Merged

Import: share assembly CCUs between projects#20296
T-Gro merged 12 commits into
dotnet:mainfrom
auduchinok:il-shareImportedAssemblies

Conversation

@auduchinok

@auduchinok auduchinok commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Two projects that reference the same assembly import it twice. Each builds its own Entity graph for every type it touches, its own EntityRefs, and for an F# assembly its own unpickling of the whole signature blob. In a solution whose projects reference the same binaries this is the single largest duplicated cost: on ten projects of ReSharper.FSharp, which reference ~500 dlls each including FSharp.Compiler.Service.dll, it is 545 MB.

An assembly's imported form can be shared only if everything it can reach is identical too, because its contents point at the CcuThunks of its own closure — and because the per-CCU caches it carries, such as CSharpStyleExtensionMembersCache, hold TyconRefs into those. So each assembly gets a key covering the file, that whole closure, and the configuration that affects how types are imported. Assemblies whose closures differ simply get separate entries.

Controlled via shareImportedAssemblies parameter in FSharpChecker.Create. Never shared: a project's own output, a type provider assembly (phase2 adds provided namespaces into its contents), a multi-module assembly (its auxiliary module loader belongs to the importing project), and anything whose closure this batch cannot account for.

Solution Projects Off On Retained
ReSharper.FSharp 10 1547.2 MB 1002.2 MB -545.0 MB (-35.2%)
Oxpecker 16 226.8 MB 190.4 MB -36.4 MB (-16.1%)
FsToolkit.ErrorHandling 8 218.3 MB 206.7 MB -11.6 MB (-5.3%)
IcedTasks 7 194.6 MB 184.5 MB -10.1 MB (-5.2%)
Fantomas 8 665.7 MB 658.3 MB -7.4 MB (-1.1%)
Prime 5 215.1 MB 212.9 MB -2.2 MB (-1.0%)
FSharp.Compiler.Service.slnx 14 2444.9 MB 2443.7 MB -1.3 MB (-0.1%)
consoleapp 1 30.9 MB 30.9 MB +0.1 MB — nothing to share

Retained memory with every project of the solution checked and held, each solution built as a real options graph so a referenced project is one node rather than a second copy. What separates the ends of that range is how much non-framework dll a solution references: sibling projects arrive as FSharpReference and are excluded by construction, and BCL assemblies are already shared by FrameworkImportsCache. ReSharper.FSharp references the compiler service and the JetBrains platform as binaries; Fantomas's own compiler fork is a project reference, so the largest thing in that repo is out of scope. Where there is nothing to share the cost is zero, not negative.

Time follows memory where the sharing is large, and is flat elsewhere. Warm, and counting ParseAndCheckProject only; median of 9 samples across processes with rotated order:

Solution Off On Time
ReSharper.FSharp 13.28 s 12.10 s -8.9%
Fantomas 3.88 s 3.90 s +0.7% — inside the spread

A cached CCU may only depend on what its key pins, so it closes over the framework layer — which the key pins by stamp, and which FrameworkImportsCache already shares — and the CCUs of its own closure, never the TcImports of whichever project imported it first. That took removing four captures: MemberSignatureEquality reaches TcGlobals through the framework layer (where GetTcGlobals was delegating anyway), and TypeForwarders, ImportProvidedType and the amap thunk that ImportILAssembly funnels every later resolution through now use an import context that resolves assembly references against the entry's closure. Verified from a heap dump: of the 81.7 MB the cache retains on Fantomas, 0.6 MB is reachable only through a TcImports, and that one is the framework layer.

Entries are held weakly. A cached CCU needs no help staying alive — the projects using it hold it through their TcImports, and it holds the rest of its own closure — so an entry lives exactly as long as it is wanted, and FSharpChecker.ClearCaches drops the lot. That closure being held strongly is what makes weak safe: no project can be handed a half-collected closure and end up with two Entity objects for one type.

@auduchinok @claude
Share the contents imported from a referenced assembly between projects
183daf4 
Two projects that reference the same assembly import it twice: each builds its own Entity graph for every
type it touches, its own EntityRefs, and for an F# assembly its own unpickling of the whole signature blob.
In a solution whose projects reference the same binaries this is the largest duplicated cost - 548 MB across
ten projects of ReSharper.FSharp, which reference ~500 dlls each.

An assembly's imported form can be shared only if everything it can reach is identical too, because its
contents point at the CcuThunks of its own closure, and because the per-CCU caches it carries - such as
CSharpStyleExtensionMembersCache - hold TyconRefs into those. So each assembly gets a key covering the file,
that whole closure, and the configuration that affects how types are imported; assemblies whose closures
differ get separate entries.

Enabled per checker with FSharpChecker.Create(shareImportedAssemblies = true), off by default, reaching the
import path as tcConfig.shareImportedAssemblies. Never shared: a project's own output, a type provider
assembly, a multi-module assembly, one whose simple name is claimed twice in a batch, and anything whose
closure the batch cannot account for.

A cached ccu may only depend on what its key pins, so it closes over the framework layer - pinned by stamp
and already shared by FrameworkImportsCache - and the ccus of its own closure, never the TcImports of
whichever project imported it first. That took removing four captures: MemberSignatureEquality now reaches
TcGlobals through the framework layer, and TypeForwarders, ImportProvidedType and the amap thunk use an
import context resolving assembly references against the entry's own closure.

Entries are held weakly. A cached ccu needs no help staying alive - the projects using it hold it through
their TcImports, and it holds the rest of its own closure - so an entry lives exactly as long as it is
wanted, and FSharpChecker.ClearCaches drops the lot. Assemblies are identified by full path and last write
time, as ILModuleReaderCacheKey does: ILAssemblyRef.QualifiedName cannot separate the builds of one package
for different target frameworks, and a solution mixing frameworks references several at once.

Retained memory, every project of the solution checked and held, each solution built as a real options graph:

  ReSharper.FSharp  10 proj  1545.3 -> 997.5 MB   -547.8 (-35.5%)
  Oxpecker          16 proj   226.9 -> 190.9 MB    -36.0 (-15.9%)
  FsToolkit          8 proj   218.3 -> 206.8 MB    -11.5  (-5.3%)
  IcedTasks          7 proj   194.4 -> 184.3 MB    -10.2  (-5.2%)
  Fantomas           8 proj   664.6 -> 657.0 MB     -7.5  (-1.1%)
  FCS solution      14 proj  2452.6 -> 2436.7 MB   -16.0  (-0.7%)
  consoleapp         1 proj    30.6 -> 30.7 MB      +0.1 - nothing to share

What separates the ends of that range is how much non-framework dll a solution references: sibling projects
arrive as FSharpReference and are out of scope here, and BCL assemblies are already shared by
FrameworkImportsCache.

Every diagnostic of every severity, sorted, is identical with sharing off and on across those solutions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok
auduchinok requested a review from a team as a code owner August 19, 2026 12:26
@github-actions

github-actions Bot commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

❗ Release notes required

You can open this PR in browser to add release notes: open in github.dev

✅ Found changes and release notes in following paths:

Change path Release notes path Description
`src/Compiler` docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

@auduchinok auduchinok changed the title Share the contents imported from a referenced assembly between projects Import: share assembly CCUs between projects Aug 19, 2026
@github-actions github-actions Bot added the AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files label Aug 19, 2026
@auduchinok
auduchinok force-pushed the il-shareImportedAssemblies branch from 0a45380 to 09a3cb1 Compare August 19, 2026 13:38
T-Gro
T-Gro reviewed Aug 24, 2026
View reviewed changes
Comment thread src/Compiler/Driver/CompilerImports.fs Outdated
T-Gro
T-Gro reviewed Aug 24, 2026
View reviewed changes
Comment thread src/Compiler/Driver/CompilerImports.fs Outdated
T-Gro
T-Gro reviewed Aug 24, 2026
View reviewed changes
Comment thread src/Compiler/Service/service.fs Outdated
T-Gro
T-Gro reviewed Aug 24, 2026
View reviewed changes
Comment thread src/Compiler/Driver/CompilerImports.fs Outdated
@github-project-automation github-project-automation Bot moved this from New to In Progress in F# Compiler and Tooling Aug 26, 2026
@auduchinok

auduchinok commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

Can this one be merged, please?

@T-Gro
T-Gro merged commit e631ed7 into dotnet:main Aug 27, 2026
52 checks passed
This was referenced Aug 28, 2026
Merged
Merged
@dotnet-maestro dotnet-maestro Bot mentioned this pull request Aug 28, 2026
Merged
auduchinok added a commit to auduchinok/fsharp that referenced this pull request Sep 1, 2026
@auduchinok @claude
Import: share a referenced project's CCU instead of pickling it
3db35d4 
A project reference pickles its signature once and every consumer unpickles its own copy: five projects
referencing one hub hold five copies of its signature TAST. The pickled tree is already in the shape
consumers need, so offer it directly through IImportedProjectCcu, and pickle only when something asks.

Four things make it usable:

- Remapped against a ccu made for this view, not the project's live one, which a consumer would else hold
  whole.
- PruneExportedSignatureInPlace brings it to the shape unpickling produces: no value definitions, no
  display-only data, no compiled-representation cache.
- Every non-local reference is re-pointed at the reading project's ccu of the same name, as unpickling
  does; a name the reader lacks keeps ours. So a consumer on another framework takes it too.
- What each name bound to is recorded, and a second consumer takes that copy only where it resolves all of
  them alike.

Binding runs once the batch is registered and before anything relinks: an assembly unpickled beside it
resolves the names it mentions against what is registered, and a delayed CCU is an error there.

Depends on sharing imported assemblies (dotnet#20296). Also fixes a check-then-act race in BackgroundCompiler:
builders were cached without a second look under the gate, so callers arriving together each built the
project.

Retained memory, under editor options. Diagnostics identical with the change off and on:

  Fantomas           8 proj  350.2 -> 275.7 MB  -74.5 (-21.3%)
  ReSharper.FSharp  10 proj  384.9 -> 304.6 MB  -80.3 (-20.9%)
  FsToolkit          8 proj   87.7 ->  70.1 MB  -17.6 (-20.1%)
  Oxpecker          16 proj  131.7 -> 114.2 MB  -17.5 (-13.3%)
  Prime              5 proj  112.2 ->  97.9 MB  -14.3 (-12.7%)
  IcedTasks          7 proj   97.0 ->  92.4 MB   -4.6  (-4.7%)
  FCS solution      14 proj  820.8 -> 818.7 MB   -2.1  (-0.2%)
  consoleapp         1 proj   29.4 ->  29.4 MB    0.0

The FCS solution gains least: nearly every edge in it crosses framework import layers, so each consumer
rebuilds a copy of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok auduchinok mentioned this pull request Sep 1, 2026
Open
auduchinok added a commit to auduchinok/fsharp that referenced this pull request Sep 2, 2026
@auduchinok @claude
Import: share a referenced project's CCU instead of pickling it
0ca6dcc 
A project reference pickles its signature once and every consumer unpickles its own copy: five projects
referencing one hub hold five copies of its signature TAST. The pickled tree is already in the shape
consumers need, so offer it directly through IImportedProjectCcu, and pickle only when something asks.

Four things make it usable:

- Remapped against a ccu made for this view, not the project's live one, which a consumer would else hold
  whole.
- PruneExportedSignatureInPlace brings it to the shape unpickling produces: no value definitions, no
  display-only data, no compiled-representation cache.
- Every non-local reference is re-pointed at the reading project's ccu of the same name, as unpickling
  does; a name the reader lacks keeps ours. So a consumer on another framework takes it too.
- What each name bound to is recorded, and a second consumer takes that copy only where it resolves all of
  them alike.

Binding runs once the batch is registered and before anything relinks: an assembly unpickled beside it
resolves the names it mentions against what is registered, and a delayed CCU is an error there.

Depends on sharing imported assemblies (dotnet#20296). Also fixes a check-then-act race in BackgroundCompiler:
builders were cached without a second look under the gate, so callers arriving together each built the
project.

Retained memory, under editor options. Diagnostics identical with the change off and on:

  Fantomas           8 proj  350.2 -> 275.7 MB  -74.5 (-21.3%)
  ReSharper.FSharp  10 proj  384.9 -> 304.6 MB  -80.3 (-20.9%)
  FsToolkit          8 proj   87.7 ->  70.1 MB  -17.6 (-20.1%)
  Oxpecker          16 proj  131.7 -> 114.2 MB  -17.5 (-13.3%)
  Prime              5 proj  112.2 ->  97.9 MB  -14.3 (-12.7%)
  IcedTasks          7 proj   97.0 ->  92.4 MB   -4.6  (-4.7%)
  FCS solution      14 proj  820.8 -> 818.7 MB   -2.1  (-0.2%)
  consoleapp         1 proj   29.4 ->  29.4 MB    0.0

The FCS solution gains least: nearly every edge in it crosses framework import layers, so each consumer
rebuilds a copy of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dotnet-bot dotnet-bot mentioned this pull request Sep 3, 2026
Closed
auduchinok added a commit to auduchinok/fsharp that referenced this pull request Sep 7, 2026
@auduchinok @claude
Import: share a referenced project's CCU instead of pickling it
6718f27 
A project reference pickles its signature once and every consumer unpickles its own copy: five projects
referencing one hub hold five copies of its signature TAST. The pickled tree is already in the shape
consumers need, so offer it directly through IImportedProjectCcu, and pickle only when something asks.

Four things make it usable:

- Remapped against a ccu made for this view, not the project's live one, which a consumer would else hold
  whole.
- PruneExportedSignatureInPlace brings it to the shape unpickling produces: no value definitions, no
  display-only data, no compiled-representation cache.
- Every non-local reference is re-pointed at the reading project's ccu of the same name, as unpickling
  does; a name the reader lacks keeps ours. So a consumer on another framework takes it too.
- What each name bound to is recorded, and a second consumer takes that copy only where it resolves all of
  them alike.

Binding runs once the batch is registered and before anything relinks: an assembly unpickled beside it
resolves the names it mentions against what is registered, and a delayed CCU is an error there.

Depends on sharing imported assemblies (dotnet#20296). Also fixes a check-then-act race in BackgroundCompiler:
builders were cached without a second look under the gate, so callers arriving together each built the
project.

Retained memory, under editor options. Diagnostics identical with the change off and on:

  Fantomas           8 proj  350.2 -> 275.7 MB  -74.5 (-21.3%)
  ReSharper.FSharp  10 proj  384.9 -> 304.6 MB  -80.3 (-20.9%)
  FsToolkit          8 proj   87.7 ->  70.1 MB  -17.6 (-20.1%)
  Oxpecker          16 proj  131.7 -> 114.2 MB  -17.5 (-13.3%)
  Prime              5 proj  112.2 ->  97.9 MB  -14.3 (-12.7%)
  IcedTasks          7 proj   97.0 ->  92.4 MB   -4.6  (-4.7%)
  FCS solution      14 proj  820.8 -> 818.7 MB   -2.1  (-0.2%)
  consoleapp         1 proj   29.4 ->  29.4 MB    0.0

The FCS solution gains least: nearly every edge in it crosses framework import layers, so each consumer
rebuilds a copy of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
auduchinok added a commit to auduchinok/fsharp that referenced this pull request Sep 9, 2026
@auduchinok @claude
Import: share a referenced project's CCU instead of pickling it
260664d 
A project reference pickles its signature once and every consumer unpickles its own copy: five projects
referencing one hub hold five copies of its signature TAST. The pickled tree is already in the shape
consumers need, so offer it directly through IImportedProjectCcu, and pickle only when something asks.

Four things make it usable:

- Remapped against a ccu made for this view, not the project's live one, which a consumer would else hold
  whole.
- PruneExportedSignatureInPlace brings it to the shape unpickling produces: no value definitions, no
  display-only data, no compiled-representation cache.
- Every non-local reference is re-pointed at the reading project's ccu of the same name, as unpickling
  does; a name the reader lacks keeps ours. So a consumer on another framework takes it too.
- What each name bound to is recorded, and a second consumer takes that copy only where it resolves all of
  them alike.

Binding runs once the batch is registered and before anything relinks: an assembly unpickled beside it
resolves the names it mentions against what is registered, and a delayed CCU is an error there.

Depends on sharing imported assemblies (dotnet#20296). Also fixes a check-then-act race in BackgroundCompiler:
builders were cached without a second look under the gate, so callers arriving together each built the
project.

Retained memory, under editor options. Diagnostics identical with the change off and on:

  Fantomas           8 proj  350.2 -> 275.7 MB  -74.5 (-21.3%)
  ReSharper.FSharp  10 proj  384.9 -> 304.6 MB  -80.3 (-20.9%)
  FsToolkit          8 proj   87.7 ->  70.1 MB  -17.6 (-20.1%)
  Oxpecker          16 proj  131.7 -> 114.2 MB  -17.5 (-13.3%)
  Prime              5 proj  112.2 ->  97.9 MB  -14.3 (-12.7%)
  IcedTasks          7 proj   97.0 ->  92.4 MB   -4.6  (-4.7%)
  FCS solution      14 proj  820.8 -> 818.7 MB   -2.1  (-0.2%)
  consoleapp         1 proj   29.4 ->  29.4 MB    0.0

The FCS solution gains least: nearly every edge in it crosses framework import layers, so each consumer
rebuilds a copy of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
auduchinok added a commit to auduchinok/fsharp that referenced this pull request Sep 10, 2026
@auduchinok @claude
Import: share a referenced project's CCU instead of pickling it
f8ba60f 
A project reference pickles its signature once and every consumer unpickles its own copy: five projects
referencing one hub hold five copies of its signature TAST. The pickled tree is already in the shape
consumers need, so offer it directly through IImportedProjectCcu, and pickle only when something asks.

Four things make it usable:

- Remapped against a ccu made for this view, not the project's live one, which a consumer would else hold
  whole.
- PruneExportedSignatureInPlace brings it to the shape unpickling produces: no value definitions, no
  display-only data, no compiled-representation cache.
- Every non-local reference is re-pointed at the reading project's ccu of the same name, as unpickling
  does; a name the reader lacks keeps ours. So a consumer on another framework takes it too.
- What each name bound to is recorded, and a second consumer takes that copy only where it resolves all of
  them alike.

Binding runs once the batch is registered and before anything relinks: an assembly unpickled beside it
resolves the names it mentions against what is registered, and a delayed CCU is an error there.

Depends on sharing imported assemblies (dotnet#20296). Also fixes a check-then-act race in BackgroundCompiler:
builders were cached without a second look under the gate, so callers arriving together each built the
project.

Retained memory, under editor options. Diagnostics identical with the change off and on:

  Fantomas           8 proj  350.2 -> 275.7 MB  -74.5 (-21.3%)
  ReSharper.FSharp  10 proj  384.9 -> 304.6 MB  -80.3 (-20.9%)
  FsToolkit          8 proj   87.7 ->  70.1 MB  -17.6 (-20.1%)
  Oxpecker          16 proj  131.7 -> 114.2 MB  -17.5 (-13.3%)
  Prime              5 proj  112.2 ->  97.9 MB  -14.3 (-12.7%)
  IcedTasks          7 proj   97.0 ->  92.4 MB   -4.6  (-4.7%)
  FCS solution      14 proj  820.8 -> 818.7 MB   -2.1  (-0.2%)
  consoleapp         1 proj   29.4 ->  29.4 MB    0.0

The FCS solution gains least: nearly every edge in it crosses framework import layers, so each consumer
rebuilds a copy of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
auduchinok added a commit to auduchinok/fsharp that referenced this pull request Sep 11, 2026
@auduchinok @claude
Import: share a referenced project's CCU instead of pickling it
cf7b36e 
A project reference pickles its signature once and every consumer unpickles its own copy: five projects
referencing one hub hold five copies of its signature TAST. The pickled tree is already in the shape
consumers need, so offer it directly through IImportedProjectCcu, and pickle only when something asks.

Four things make it usable:

- Remapped against a ccu made for this view, not the project's live one, which a consumer would else hold
  whole.
- PruneExportedSignatureInPlace brings it to the shape unpickling produces: no value definitions, no
  display-only data, no compiled-representation cache.
- Every non-local reference is re-pointed at the reading project's ccu of the same name, as unpickling
  does; a name the reader lacks keeps ours. So a consumer on another framework takes it too.
- What each name bound to is recorded, and a second consumer takes that copy only where it resolves all of
  them alike.

Binding runs once the batch is registered and before anything relinks: an assembly unpickled beside it
resolves the names it mentions against what is registered, and a delayed CCU is an error there.

Depends on sharing imported assemblies (dotnet#20296). Also fixes a check-then-act race in BackgroundCompiler:
builders were cached without a second look under the gate, so callers arriving together each built the
project.

Retained memory, under editor options. Diagnostics identical with the change off and on:

  Fantomas           8 proj  350.2 -> 275.7 MB  -74.5 (-21.3%)
  ReSharper.FSharp  10 proj  384.9 -> 304.6 MB  -80.3 (-20.9%)
  FsToolkit          8 proj   87.7 ->  70.1 MB  -17.6 (-20.1%)
  Oxpecker          16 proj  131.7 -> 114.2 MB  -17.5 (-13.3%)
  Prime              5 proj  112.2 ->  97.9 MB  -14.3 (-12.7%)
  IcedTasks          7 proj   97.0 ->  92.4 MB   -4.6  (-4.7%)
  FCS solution      14 proj  820.8 -> 818.7 MB   -2.1  (-0.2%)
  consoleapp         1 proj   29.4 ->  29.4 MB    0.0

The FCS solution gains least: nearly every edge in it crosses framework import layers, so each consumer
rebuilds a copy of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@T-Gro T-Gro T-Gro approved these changes

Assignees

No one assigned

Labels

AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files

Projects

F# Compiler and Tooling
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@auduchinok @T-Gro