fix(throttling-manager): Apply throttling behind a tandem and with retry_on_blocked disabled

[Mantisus]

Description

• Recording a 429 is now separate from the session blocking check and runs regardless of retry_on_blocked, and the crawler resolves its ThrottlingRequestManager through any RequestManagerTandem around it. Both configurations previously dropped the 429 backoff and the robots.txt crawl-delay, while warning that no throttler was in use.

Issues

• Closes: ThrottlingRequestManager silently does nothing in two configurations #2126

Testing

• Added new unit tests

[janbuchar]

Hi @Mantisus and thanks for the PR! I did a bunch more research in the JS port and figured out a different way around this. It involves changes to the request manager, so it will presumably take more time to pass code review.

Just letting you know so that you don't end up thinking I forgot about this 🙂

[janbuchar]

@Mantisus feel free to check apify/crawlee#4061 now — in its current state, it should be pretty clear what direction it's headed.

However, it's unlikely that it can be replicated here without breaking changes. @vdusek do you think it makes sense to wait for v2, or should we try and think of a backwards-compatible fix?

[vdusek]

@Mantisus feel free to check apify/crawlee#4061 now — in its current state, it should be pretty clear what direction it's headed.

However, it's unlikely that it can be replicated here without breaking changes. @vdusek do you think it makes sense to wait for v2, or should we try and think of a backwards-compatible fix?

If we manage to make it backward-compatible in a reasonable way, I'd go for it. But if that requires introducing some weird quirks/workarounds, I would wait for next major. Especially considering that, nobody has actually reported this issue, if I'm correct.