v2.6.0: structured errors — translate non-2xx at the engine

CHANGELOG.md

@@ -1,5 +1,24 @@
 # Bright Data Python SDK Changelog
 
+## Version 2.6.0 - Structured errors
+
+- **Every non-2xx is now a typed exception.** `AsyncEngine` previously converted only 401/403 and handed every other failure back as a normal response, leaving each subsystem to improvise. Classification now happens once, at the single point every request passes through: `429` → the new **`RateLimitError`** (with `retry_after` parsed from the header), other 4xx/5xx → `APIError`.
+- **Exceptions carry data, not just prose.** `BrightDataError` gained `status_code`, `url`, `method`, `retry_after`, `retryable` and `raw` (the response body, bounded to 4 KB). Callers can branch on the failure instead of parsing its message.
+- **Fixed**: a rate-limited dataset request surfaced as an aiohttp *content-type* error, because the 429 body is a bare string and the response was parsed as JSON before its status was checked. It now raises `RateLimitError`.
+- **Fixed**: a token expiring mid-poll was reported as `"Job failed with status: error"` — blaming the user's scrape for an authentication problem. `DatasetAPIClient.get_status` no longer collapses every non-200 into the status string `"error"`.
+- **Results carry the failure too.** `ScrapeResult` / `CrawlResult` gained `cause`, the originating exception, populated wherever one is converted into a result. `error` remains the human-readable message:
+  ```python
+  result = await client.scrape.x.posts(url)
+  if not result.success and isinstance(result.cause, RateLimitError):
+      await asyncio.sleep(result.cause.retry_after or 60)
+  ```
+- **Retry is now opt-in.** `retry_with_backoff` no longer retries by exception type. An error with no status code is raised locally — sometimes *after* the server accepted the work — so repeating it could create a duplicate billed job; those are never retried. Explicit 5xx still is. **429 never is**, because those responses consume quota and retrying extends the lockout.
+- `DatasetError` now subclasses `BrightDataError` (still catchable as before). `RateLimitError` and `DataNotReadyError` are exported from `brightdata` top level.
+- **Note on messages**: error messages are shorter and more uniform, with detail moved to attributes. Code matching on message *text* may need updating; use `status_code` instead.
+- **Known limit**: this is status-based, so it cannot see HTTP 200 responses carrying an error in the body (SERP's inner envelope, Web Unlocker error content). Those remain string-shaped.
+
+---
+
 ## Version 2.4.0 - Sync parity, colorless job verbs, dataset error reporting
 
 - **Sync client parity**: `SyncBrightDataClient` now mirrors the async surface. Added `client.datasets` (fixes the `SyncBrightDataClient` `datasets` `AttributeError`), the 5 missing scrapers (`scrape.tiktok` / `youtube` / `reddit` / `perplexity` / `digikey`), the 2 missing search verticals (`search.tiktok` / `youtube`), Pinterest trigger/status/fetch, and Instagram-search `profiles` / `reels_all`.

README.md

@@ -22,6 +22,34 @@ export BRIGHTDATA_API_TOKEN="your_api_token_here"
 **Already logged in with the CLI?** The SDK works with no configuration — it automatically
 falls back to the credentials stored by `brightdata login`.
 
+## Handling errors
+
+Failures carry structured data, not just a message:
+
+```python
+from brightdata import BrightDataClient, RateLimitError, APIError
+
+async with BrightDataClient() as client:
+    try:
+        data = await client.datasets.instagram_profiles.download(snapshot_id)
+    except RateLimitError as e:
+        await asyncio.sleep(e.retry_after or 60)   # the API told us how long
+    except APIError as e:
+        print(e.status_code, e.raw)                # not a message to parse
+```
+
+Methods that return a result instead of raising expose the same information on `cause`:
+
+```python
+result = await client.scrape.x.posts(url)
+if not result.success and isinstance(result.cause, RateLimitError):
+    await asyncio.sleep(result.cause.retry_after or 60)
+```
+
+Use `e.retryable` to decide whether repeating the call is safe. Rate limits are never
+retryable — a 429 response itself consumes quota, so retrying extends the lockout; wait
+`retry_after` instead.
+
 ## Quick Start
 
 This SDK is **async-native**. A sync client is also available (see [Sync Client](#sync-client)).

pyproject.toml

@@ -7,7 +7,7 @@ where = ["src"]
 
 [project]
 name = "brightdata-sdk"
-version = "2.5.0"
+version = "2.6.0"
 description = "Modern async-first Python SDK for Bright Data APIs"
 authors = [{name = "Bright Data", email = "support@brightdata.com"}]
 license = {text = "MIT"}

src/brightdata/__init__.py

@@ -69,6 +69,8 @@
     ValidationError,
     AuthenticationError,
     APIError,
+    RateLimitError,
+    DataNotReadyError,
     ZoneError,
     NetworkError,
     SSLError,
@@ -127,6 +129,8 @@
     "ValidationError",
     "AuthenticationError",
     "APIError",
+    "RateLimitError",
+    "DataNotReadyError",
     "ZoneError",
     "NetworkError",
     "SSLError",

src/brightdata/core/engine.py

@@ -6,9 +6,16 @@
 import warnings
 from typing import Optional, Dict, Any
 from .. import __version__
-from ..exceptions import AuthenticationError, NetworkError, SSLError
+from ..exceptions import (
+    APIError,
+    AuthenticationError,
+    NetworkError,
+    RateLimitError,
+    SSLError,
+)
 from http import HTTPStatus
 from ..utils.ssl_helpers import is_ssl_certificate_error, get_ssl_error_message
+from ..utils.http import parse_retry_after, status_phrase
 
 # Rate limiting support
 try:
@@ -409,19 +416,43 @@ async def __aenter__(self):
                         headers=self._headers,
                         timeout=self._timeout,
                     )
-                    # Check status codes that should raise exceptions
-                    if self._response.status == HTTPStatus.UNAUTHORIZED:
-                        text = await self._response.text()
-                        await self._response.release()
-                        raise AuthenticationError(
-                            f"Unauthorized ({HTTPStatus.UNAUTHORIZED}): {text}"
+                    status = self._response.status
+
+                    # 202 MUST pass through. It is a success for
+                    # scraper_studio.trigger_immediate, and elsewhere it means
+                    # "ready but still building", which fetch_result turns into
+                    # DataNotReadyError -- the SDK's only recovery path.
+                    if status < 400 or status == HTTPStatus.ACCEPTED:
+                        return self._response
+
+                    text = await self._response.text()
+                    await self._response.release()
+
+                    context = {
+                        "status_code": status,
+                        "url": self._url,
+                        "method": self._method,
+                        "raw": text,
+                    }
+                    # A short body excerpt keeps a bare print(exc) useful; the
+                    # full (bounded) body stays on .raw.
+                    detail = " ".join(text.split())[:200]
+                    suffix = f": {detail}" if detail else ""
+
+                    if status in (HTTPStatus.UNAUTHORIZED, HTTPStatus.FORBIDDEN):
+                        raise AuthenticationError(f"{status_phrase(status)} ({status})", **context)
+
+                    if status == HTTPStatus.TOO_MANY_REQUESTS:
+                        # Never retryable: a 429 response itself consumes quota,
+                        # so retrying extends the lockout instead of waiting it out.
+                        raise RateLimitError(
+                            f"Rate limited ({status}){suffix}",
+                            retry_after=parse_retry_after(self._response.headers),
+                            **context,
                         )
-                    elif self._response.status == HTTPStatus.FORBIDDEN:
-                        text = await self._response.text()
-                        await self._response.release()
-                        raise AuthenticationError(f"Forbidden ({HTTPStatus.FORBIDDEN}): {text}")
 
-                    return self._response
+                    # retryable resolves from status_code: 5xx yes, 4xx no.
+                    raise APIError(f"Request failed (HTTP {status}){suffix}", **context)
                 except asyncio.TimeoutError as e:
                     # Must be caught before OSError — on Python 3.11+,
                     # TimeoutError is a subclass of OSError

src/brightdata/crawler/models.py

@@ -4,6 +4,8 @@
 from datetime import datetime
 from typing import Any, Dict, List, Optional
 
+from ..exceptions import BrightDataError
+
 
 @dataclass
 class CrawlResult:
@@ -22,6 +24,9 @@ class CrawlResult:
     trigger_sent_at: Optional[datetime] = None
     data_fetched_at: Optional[datetime] = None
     error: Optional[str] = None
+    # Underlying exception when one was converted into this result; `error`
+    # stays the message, `cause` is what code branches on.
+    cause: Optional[BrightDataError] = field(default=None, repr=False, compare=False)
 
     def __repr__(self) -> str:
         sid = f" snapshot_id={self.snapshot_id}" if self.snapshot_id else ""

src/brightdata/crawler/service.py

@@ -22,7 +22,7 @@
 from typing import Any, Dict, List, Optional, TYPE_CHECKING, Union
 
 from .models import CrawlJob, CrawlResult
-from ..exceptions import APIError, ValidationError
+from ..exceptions import APIError, BrightDataError, ValidationError
 from ..utils.function_detection import get_caller_function_name
 from ..utils.validation import validate_url, validate_url_list
 
@@ -222,6 +222,7 @@ async def download(
                     trigger_sent_at=trigger_sent_at,
                     data_fetched_at=datetime.now(timezone.utc),
                     error=f"Status check failed: {exc}",
+                    cause=exc,
                 )
 
             if current == "ready":
@@ -298,8 +299,19 @@ async def _scrape_sync(
                     trigger_sent_at=trigger_sent_at,
                     data_fetched_at=data_fetched_at,
                 )
-        except (ValidationError, APIError):
+        except ValidationError:
             raise
+        except APIError as exc:
+            # The engine now raises for non-2xx, so this is the same condition
+            # the status check above used to handle inline. Keep returning a
+            # CrawlResult rather than raising, which is crawl()'s contract.
+            return CrawlResult(
+                success=False,
+                trigger_sent_at=trigger_sent_at,
+                data_fetched_at=datetime.now(timezone.utc),
+                error=f"HTTP {exc.status_code}: {exc.raw or exc.message}",
+                cause=exc,
+            )
         except Exception as exc:
             return CrawlResult(
                 success=False,
@@ -347,6 +359,7 @@ async def _fetch_snapshot(
                 trigger_sent_at=trigger_sent_at,
                 data_fetched_at=datetime.now(timezone.utc),
                 error=f"Snapshot fetch error: {exc}",
+                cause=exc if isinstance(exc, BrightDataError) else None,
             )
 
     @staticmethod

src/brightdata/datasets/base.py

@@ -3,21 +3,44 @@
 """
 
 import asyncio
+import json
 import time
 from typing import Dict, List, Any, Optional, Literal, TYPE_CHECKING
 
 from .models import DatasetMetadata, SnapshotStatus
+from ..exceptions import APIError, BrightDataError, RateLimitError
 
 if TYPE_CHECKING:
     from ..core.engine import AsyncEngine
 
 
-class DatasetError(Exception):
+class DatasetError(BrightDataError):
     """Error related to dataset operations."""
 
     pass
 
 
+def _as_dataset_error(exc: APIError, what: str) -> DatasetError:
+    """
+    Re-type an engine-raised APIError as a DatasetError, preserving context.
+
+    The engine classifies every non-2xx centrally, so by the time a failure
+    reaches this layer it is already structured. Callers of the datasets API
+    catch DatasetError, though, so convert rather than let a sibling type
+    escape. RateLimitError is deliberately NOT converted: it is the more
+    specific, actionable type and users are told to catch it directly.
+    """
+    return DatasetError(
+        f"{what} failed (HTTP {exc.status_code})",
+        status_code=exc.status_code,
+        url=exc.url,
+        method=exc.method,
+        retry_after=exc.retry_after,
+        retryable=exc.retryable,
+        raw=exc.raw,
+    )
+
+
 class BaseDataset:
     """
     Base class for all dataset types.
@@ -95,11 +118,17 @@ async def __call__(
         if records_limit is not None:
             payload["records_limit"] = records_limit
 
-        async with self._engine.post_to_url(
-            f"{self.BASE_URL}/datasets/filter",
-            json_data=payload,
-        ) as response:
-            data = await response.json()
+        try:
+            async with self._engine.post_to_url(
+                f"{self.BASE_URL}/datasets/filter",
+                json_data=payload,
+            ) as response:
+                body = await response.text()
+                data = json.loads(body) if body.strip() else {}
+        except RateLimitError:
+            raise
+        except APIError as exc:
+            raise _as_dataset_error(exc, "Filter request") from exc
 
         if "snapshot_id" not in data:
             error_msg = (
@@ -142,10 +171,16 @@ async def get_status(self, snapshot_id: str) -> SnapshotStatus:
         Returns:
             SnapshotStatus with status field: "scheduled", "building", "ready", or "failed"
         """
-        async with self._engine.get_from_url(
-            f"{self.BASE_URL}/datasets/snapshots/{snapshot_id}"
-        ) as response:
-            data = await response.json()
+        try:
+            async with self._engine.get_from_url(
+                f"{self.BASE_URL}/datasets/snapshots/{snapshot_id}"
+            ) as response:
+                body = await response.text()
+                data = json.loads(body) if body.strip() else {}
+        except RateLimitError:
+            raise
+        except APIError as exc:
+            raise _as_dataset_error(exc, "Snapshot status check") from exc
         return SnapshotStatus.from_dict(data)
 
     async def download(
@@ -197,8 +232,6 @@ async def download(
             f"{self.BASE_URL}/datasets/snapshots/{snapshot_id}/download",
             params={"format": format},
         ) as response:
-            import json
-
             # Check for HTTP errors
             if response.status >= 400:
                 error_text = await response.text()

src/brightdata/exceptions/__init__.py

@@ -5,6 +5,7 @@
     ValidationError,
     AuthenticationError,
     APIError,
+    RateLimitError,
     DataNotReadyError,
     ZoneError,
     NetworkError,
@@ -16,6 +17,7 @@
     "ValidationError",
     "AuthenticationError",
     "APIError",
+    "RateLimitError",
     "DataNotReadyError",
     "ZoneError",
     "NetworkError",