feat: snapshot the station at the end of a measurement and diff snapshots

docs/changes/newsfragments/8501.new

@@ -0,0 +1,18 @@
+In addition to the snapshot taken when a measurement starts, QCoDeS now also
+takes a snapshot of the station when the measurement ends. The end snapshot is
+taken with the same settings as the start snapshot (``update="Only_invalid"``)
+and is stored in the metadata of the dataset under the ``end_snapshot`` key. It
+is available as the new ``end_snapshot`` property of the dataset. Snapshotting
+at the end can be disabled for a single measurement by passing
+``snapshot_at_end=False`` to ``Measurement.run``, or globally via the new
+``snapshot_at_end`` key in the ``dataset`` section of the QCoDeS config.
+
+Two new functions, ``qcodes.dataset.diff_start_end_snapshot`` and
+``qcodes.dataset.diff_start_end_snapshot_by_id``, return the differences
+between the parameter values of the start and the end snapshot of a dataset as
+a ``ParameterDiff``, complementing the existing
+``qcodes.dataset.diff_param_snapshots`` and
+``qcodes.dataset.diff_param_values_by_id`` which compare two datasets and which
+are now exported from ``qcodes.dataset``. A ``ParameterDiff`` can be rendered in
+a human-readable form with the new ``qcodes.utils.format_parameter_diff``, which
+is also used when printing a ``ParameterDiff``.

docs/examples/DataSet/Working with snapshots.ipynb

@@ -884,12 +884,109 @@
     "diff_param_values(dataset.snapshot, bad_dataset.snapshot).changed"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Snapshot at the end of a measurement\n",
+    "\n",
+    "In addition to the snapshot taken when a measurement starts, QCoDeS takes a second snapshot when the measurement ends. This makes it possible to see how the state of the setup evolved during the measurement itself.\n",
+    "\n",
+    "The end snapshot is taken with the same settings as the start snapshot (`update=\"Only_invalid\"`) and is stored in the metadata of the dataset under the `end_snapshot` key. It is available as the `end_snapshot` property of the dataset, which returns a python dictionary (or `None` if no end snapshot was taken).\n",
+    "\n",
+    "Snapshotting at the end can be disabled for a single measurement by passing `snapshot_at_end=False` to `Measurement.run`, or globally via the `snapshot_at_end` key in the `dataset` section of the QCoDeS config."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "measurement = Measurement(experiment, station)\n",
+    "\n",
+    "measurement.register_parameter(instr.input)\n",
+    "measurement.register_parameter(instr.output, setpoints=[instr.input])\n",
+    "\n",
+    "instr.gain(11)\n",
+    "\n",
+    "with measurement.run() as data_saver:\n",
+    "    input_value = 111\n",
+    "    instr.input.set(input_value)\n",
+    "    instr.output.set(222)\n",
+    "    data_saver.add_result((instr.input, input_value), (instr.output, instr.output()))\n",
+    "    # the gain drifts (or is changed) while the measurement is running\n",
+    "    instr.gain(42)\n",
+    "\n",
+    "dataset_with_end_snapshot = data_saver.dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(\n",
+    "    dataset_with_end_snapshot.end_snapshot[\"station\"][\"instruments\"][\"instr\"][\n",
+    "        \"parameters\"\n",
+    "    ][\"gain\"]\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Diffing the start and the end snapshot\n",
+    "\n",
+    "`diff_start_end_snapshot` returns the differences between the parameter values of the two snapshots of a single dataset as a `ParameterDiff`, in the same way as `diff_param_values` does for two separate snapshots. `diff_start_end_snapshot_by_id` does the same given the run id of a dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from qcodes.dataset import diff_start_end_snapshot, diff_start_end_snapshot_by_id\n",
+    "\n",
+    "diff = diff_start_end_snapshot(dataset_with_end_snapshot)\n",
+    "diff.changed"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A `ParameterDiff` can be rendered in a human-readable form with `format_parameter_diff`, which also allows naming the two sides of the diff. Printing a `ParameterDiff` directly gives the same rendering with the default names."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from qcodes.utils import format_parameter_diff\n",
+    "\n",
+    "print(format_parameter_diff(diff, \"start\", \"end\"))"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "print(\n",
+    "    format_parameter_diff(\n",
+    "        diff_start_end_snapshot_by_id(dataset_with_end_snapshot.run_id),\n",
+    "        \"start\",\n",
+    "        \"end\",\n",
+    "    )\n",
+    ")"
+   ]
   }
  ],
  "metadata": {

src/qcodes/configuration/qcodesrc.json

@@ -79,7 +79,8 @@
         "export_chunked_export_of_large_files_enabled": false,
         "export_chunked_threshold": 1000,
         "in_memory_cache": true,
-        "load_from_exported_file": false
+        "load_from_exported_file": false,
+        "snapshot_at_end": true
     },
     "telemetry":
     {

src/qcodes/configuration/qcodesrc_schema.json

@@ -382,6 +382,11 @@
                     "type": "boolean",
                     "default": true,
                     "description": "Should the data be cached in memory as it is measured. Useful to disable for large datasets to save on memory consumption."
+                },
+                "snapshot_at_end": {
+                    "type": "boolean",
+                    "default": true,
+                    "description": "Should a snapshot of the station be taken at the end of a measurement in addition to the one taken at the start. The end snapshot is stored in the dataset metadata under the 'end_snapshot' key."
                 }
             },
             "description": "Settings related to the DataSet and Measurement Context manager",

src/qcodes/dataset/__init__.py

@@ -47,6 +47,12 @@
 )
 from .measurements import Measurement
 from .plotting import plot_by_id, plot_dataset
+from .snapshot_utils import (
+    diff_param_snapshots,
+    diff_param_values_by_id,
+    diff_start_end_snapshot,
+    diff_start_end_snapshot_by_id,
+)
 from .sqlite.connection import (
     AtomicConnection,
 )
@@ -88,6 +94,10 @@
     "call_params_threaded",
     "connect",
     "datasaver_builder",
+    "diff_param_snapshots",
+    "diff_param_values_by_id",
+    "diff_start_end_snapshot",
+    "diff_start_end_snapshot_by_id",
     "do0d",
     "do1d",
     "do2d",

src/qcodes/dataset/data_set_protocol.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import json
 import logging
 import os
 import warnings
@@ -69,6 +70,11 @@
 
 LOG = logging.getLogger(__name__)
 
+# TODO(jenshnielsen): Consider adding a dedicated ``end_snapshot`` column to the
+# runs table via a database upgrade rather than storing the end snapshot as
+# metadata in a dynamic column.
+END_SNAPSHOT_METADATA_KEY = "end_snapshot"
+
 
 class CompletedError(RuntimeError):
     pass
@@ -168,6 +174,14 @@ def add_snapshot(self, snapshot: str, overwrite: bool = False) -> None: ...
     @property
     def _snapshot_raw(self) -> str | None: ...
 
+    @property
+    def end_snapshot(self) -> dict[str, Any] | None: ...
+
+    def add_end_snapshot(self, snapshot: str, overwrite: bool = False) -> None: ...
+
+    @property
+    def _end_snapshot_raw(self) -> str | None: ...
+
     def add_metadata(self, tag: str, metadata: Any) -> None: ...
 
     @property
@@ -547,6 +561,51 @@ def dependent_parameters(self) -> tuple[ParamSpecBase, ...]:
         """
         return tuple(self.description.interdeps.dependencies.keys())
 
+    @property
+    def end_snapshot(self) -> dict[str, Any] | None:
+        """
+        Snapshot taken at the end of the run as a dictionary (or None if no
+        such snapshot was taken).
+        """
+        snapshot_json = self._end_snapshot_raw
+        if snapshot_json is not None:
+            return json.loads(snapshot_json)
+        else:
+            return None
+
+    @property
+    def _end_snapshot_raw(self) -> str | None:
+        """
+        Snapshot taken at the end of the run as a JSON-formatted string
+        (or None).
+        """
+        snapshot_raw = self.metadata.get(END_SNAPSHOT_METADATA_KEY)
+        if snapshot_raw is None:
+            return None
+        if not isinstance(snapshot_raw, str):
+            raise TypeError(
+                f"Expected the end snapshot of run {self.guid} to be a string "
+                f"but got {type(snapshot_raw)}."
+            )
+        return snapshot_raw
+
+    def add_end_snapshot(self, snapshot: str, overwrite: bool = False) -> None:
+        """
+        Add a snapshot taken at the end of the run to this dataset.
+
+        Args:
+            snapshot: the raw JSON dump of the snapshot
+            overwrite: force overwrite an existing end snapshot
+
+        """
+        if self._end_snapshot_raw is None or overwrite:
+            self.add_metadata(END_SNAPSHOT_METADATA_KEY, snapshot)
+        else:
+            LOG.warning(
+                "This dataset already has an end snapshot. "
+                "Use overwrite=True to overwrite that"
+            )
+
 
 class DataSetType(StrEnum):
     DataSet = "DataSet"