Migrated website from python to nodejs and merged HAR docs

[max-ostapenko]

• all the httparchive.org content managed by Astro
• most of the website is served as static, moved dates and metrics to API endpoints
• merged har.fyi content under docs/

There were many duplicated and custom operations, like JS management and resources versioning, that are taken care of by framework anyway.

[max-ostapenko]

@tunetheweb closed all TODO's.

[max-ostapenko]

@tunetheweb here are the summarized changes.

Preview deployment: https://httparchive--staging-supndlqf.web.app/
Hosting console: https://console.firebase.google.com/project/httparchive/hosting/sites/httparchive/channels/staging

1. Hosting

• Migrated from App Engine to Firebase Hosting: Converted legacy Python/Flask App Engine setup to a modern, serverless static architecture powered by Astro + Firebase Hosting.
• Removed App Engine Code: Deleted app.yaml, server.js, Python server/ files, Jinja templates/, and express dependency.
• Added Firebase Hosting Configs:
  • .firebaserc: Configured project mapping to httparchive.
  • firebase.json: Configured static serving out of dist/, cleanUrls: true, trailingSlash: false, font caching (/static/fonts/**), security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy), and default emulator port 8080.

2. Custom 404 & SEO Parity

• Custom 404 Page: Added src/pages/404.astro matching original App Engine 404 template parity (issue link, nav buttons, magnifying glass illustration).
• Starlight Config: Set disable404Route: true in astro.config.mjs so Astro compiles the custom 404 page directly to dist/404.html.
• Enhanced SEO Timestamps: Updated src/layouts/Base.astro to apply last_updated.json date_published and date_modified metadata across all /docs/* routes.

3. Development Workflow

• npm start: Builds site and starts native Firebase Hosting emulator (npx firebase-tools emulators:start --only hosting).
• npm run stage: Deploys to Firebase Hosting staging preview channel (npx firebase-tools hosting:channel:deploy staging).
• npm run deploy: Deploys to Firebase Hosting live production channel (npx firebase-tools deploy --only hosting).
• Removed Redundant Scripts: Removed tools/scripts/deploy.sh and obsolete preview / testdeploy npm commands.

4. GitHub Actions

• firebase-preview.yml: Added workflow to automatically build and deploy preview channels for all incoming Pull Requests using FirebaseExtended/action-hosting-deploy@v0.
• deploy.yml: Updated workflow to deploy to Firebase Hosting live channel on push to main.
• Automated Test Scripts: Updated tools/scripts/run_and_test_website.sh and run_and_test_website.bat to run tests against the Firebase Hosting emulator (100% test pass rate: 54/54 passed).
• Lighthouse CI: Updated tools/scripts/set_lighthouse_urls.sh diff paths to scan src/pages/ and public/sitemap.xml.

5. Code Generators (tools/generate/)

• Extracted Shared Helpers: Moved map_to_astro_page() to tools/generate/shared.js.
• Removed Dead Code & Dependencies: Uninstalled recursive-readdir and removed dead find_asset_files() function.
• Modernized Node APIs: Replaced recursive-readdir with native fs.promises.readdir({ recursive: true }) in generate_timestamps.js and replaced deprecated .substr() in generate_sitemap.js.
• Cleaned last_updated.json: Pruned legacy /static/ asset entries from config/last_updated.json (reduced lines from 403 to 203), focusing purely on content pages, reports, and documentation.

6. UI

• Subdomain Lens Detection: Added client-side subdomain detection in src/pages/reports/[report_id].astro and src/js/report.js so subdomains (e.g. wordpress.httparchive.org) automatically pre-select their corresponding lens.

7. Route Mapping (Flask vs. Astro)

Route Path	Production main (Flask)	Migrated Version (Astro / Firebase)	Output Artifact
/	server/routes.py: index() -> templates/index.html	src/pages/index.astro	dist/index.html
/about	server/routes.py: about() -> templates/about.html	src/pages/about.astro	dist/about/index.html
/faq	server/routes.py: faq() -> templates/faq.html	src/pages/faq.astro	dist/faq/index.html
/reports	server/routes.py: reports() -> templates/reports.html	src/pages/reports/index.astro	dist/reports/index.html
/reports/[report_id]	server/routes.py: report() -> templates/report/report.html	src/pages/reports/[report_id].astro	dist/reports/[report_id]/index.html
/reports/techreport/landing	server/routes.py: techreportlanding() -> templates/techreport/landing.html	src/pages/reports/techreport/landing.astro	dist/reports/techreport/landing/index.html
/reports/techreport/tech	server/routes.py: techreport() -> templates/techreport/techreport.html	src/pages/reports/techreport/tech.astro	dist/reports/techreport/tech/index.html
/reports/techreport/[page_id]	server/routes.py: techreportlanding()	src/pages/reports/techreport/[page_id].astro	dist/reports/techreport/[page_id]/index.html
/docs/*	N/A	src/content/docs/docs/* (Starlight MDX)	dist/docs/*
/404	server/errors.py -> templates/error/404.html	src/pages/404.astro	dist/404.html
/sitemap.xml	server/routes.py -> templates/sitemap.xml	tools/generate/generate_sitemap.js	public/sitemap.xml
/robots.txt	server/routes.py: static_from_root()	public/robots.txt	dist/robots.txt
/favicon.ico	server/routes.py: favicon()	public/favicon.ico	dist/favicon.ico

---

8. HTTP Headers

Production Talisman headers (server/csp.py) were ported 1:1 into firebase.json for all routes (/**).

9. HAR

Migrated into content/docs/docs.

10. Server logic moved to API

See HTTPArchive/tech-report-apis#128

Note: I think it may be more ergonomic to merge API here, keeping closer to frontend.

[tunetheweb]

@max-ostapenko finally coming round to look at this (apologies for the delay!) and stumbling at the first hurdle:

$ npm i
npm error code ERESOLVE
npm error ERESOLVE could not resolve
npm error
npm error While resolving: starlight-llms-txt@0.10.0
npm error Found: astro@7.0.3
npm error node_modules/astro
npm error   dev astro@"^7.0.3" from the root project
npm error   peer astro@"^7.0.0-alpha.0" from @astrojs/mdx@7.0.0
npm error   node_modules/@astrojs/mdx
npm error     @astrojs/mdx@"^7.0.0" from @astrojs/starlight@0.41.1
npm error     node_modules/@astrojs/starlight
npm error       dev @astrojs/starlight@"^0.41.1" from the root project
npm error       2 more (starlight-links-validator, starlight-llms-txt)
npm error   3 more (@astrojs/starlight, astro-expressive-code, starlight-links-validator)
npm error
npm error Could not resolve dependency:
npm error peer astro@"^6.0.0" from starlight-llms-txt@0.10.0
npm error node_modules/starlight-llms-txt
npm error   dev starlight-llms-txt@"^0.10.0" from the root project
npm error
npm error Conflicting peer dependency: astro@6.4.8
npm error node_modules/astro
npm error   peer astro@"^6.0.0" from starlight-llms-txt@0.10.0
npm error   node_modules/starlight-llms-txt
npm error     dev starlight-llms-txt@"^0.10.0" from the root project
npm error
npm error Fix the upstream dependency conflict, or retry this command with --force or --legacy-peer-deps to accept an incorrect (and potentially broken) dependency resolution.
npm error
npm error

I also see some merge conflicts.

Can you look at resolving this and will have a look.

[max-ostapenko]

@tunetheweb resolved.

[tunetheweb]

Not gone through it all, but going to stop here given the issues with the /docs site. As started in the other comment, I think we should split this PR.

I also see some styling differences:

[tunetheweb]

Out of interest why are these not needed? They are used to provide a lastmod to /sitemap.xml

[max-ostapenko]

I was testing automating this step, and forgot to add it back.

[tunetheweb]

Suggested change

The test agents are run from [Google Cloud regions](https://cloud.google.com/compute/docs/regions-zones) across the US. Each URL is loaded once with an empty cache ("first view") for normal metrics collection and again, in a clean browser profile, using [Lighthouse](https://developers.google.com/web/tools/lighthouse). The data is collected via a [HAR file](https://en.wikipedia.org/wiki/.har). The HTTP Archive collects these HAR files, parses them, and populates [a public dataset in BigQuery](docs/guides/getting-started/).

The test agents are run from [Google Cloud regions](https://cloud.google.com/compute/docs/regions-zones) across the US. Each URL is loaded once with an empty cache ("first view") for normal metrics collection and again, in a clean browser profile, using [Lighthouse](https://developers.google.com/web/tools/lighthouse). The data is collected via a [HAR file](https://en.wikipedia.org/wiki/.har). The HTTP Archive collects these HAR files, parses them, and populates [a public dataset in BigQuery](/docs/guides/getting-started/).

[tunetheweb]

Suggested change

	The HTTP Archive dataset is available publicly on BigQuery. Check out [Getting Started Accessing the HTTP Archive with BigQuery](docs/guides/getting-started/).
	The HTTP Archive dataset is available publicly on BigQuery. Check out [Getting Started Accessing the HTTP Archive with BigQuery](/docs/guides/getting-started/).

[tunetheweb]

Why map.png instead of web.png as it was before? That's not a great name either to be honest so maybe hero.png is better?

[max-ostapenko]

renamed

[tunetheweb]

This doesn't auto refresh when editing code. Should we use astro:dev which includes a server so doesn't need firebase?

Or is there some combo of both if you want to test e2e to include firebase?

[max-ostapenko]

astro:dev definitely gives auto-refresh, not sure you can connect these 2 stages live.

[tunetheweb]

Do we really need the double nesting of docs in content/docs/docs/? I know it isn't surfaced to the user in the URL but it's still quite confusing...

[max-ostapenko]

First docs/ is required by Astro, second is just a path.
Should we use any other path for docs?

[tunetheweb]

I see a double hamburger menu in /docs:

Additionally, the original hamburger menu differs from the rest of the site (Reports is not, and cannot be expanded):

I think we're biting off too much in one go here and should split this PR into 3 PRs:

• Migrate from python to node
• Migrate from App Engine to Firebase
• Integrate har.fyi