Runtime checkability?

[nstarman]

Do we want the Protocols to be runtime checkable?

Pros:

• useful at runtime

Cons:

• @runtime_checkable has some sharp bits! Can get false positives / negatives.

[jorenham]

I prefer -> TypeIs[...] functions like #19 as part of the public API.

There's also a secret option number three: jorenham/optype#252. It's not for the faint of heart, though (see e.g. https://github.com/jorenham/optype/blob/c48c16c1e0dea630d4d99509e0f911a55e627808/optype/_core/_just.py#L93-L134).

[nstarman]

Then maybe we should standardize the naming scheme?

def has_arraynamespace(): ...  # works
def has_array(): ...  # doesn't work well

[jorenham]

Then maybe we should standardize the naming scheme?

def has_arraynamespace(): ...  # works
def has_array(): ...  # doesn't work well

Yea good idea

[NeilGirdhar]

I just want to propose an alternative for discussion. I'm not sure if it's doable, but given that we have control over array-api-compat, could we have array_namespace return an object that explicitly derives from an ABC ArrayNamespace?

Then you have perfect runtime check-ability, and you don't have to worry about keeping array-api-compat in sync with array-api-typing?

[jorenham]

I just want to propose an alternative for discussion. I'm not sure if it's doable, but given that we have control over array-api-compat, could we have array_namespace return an object that explicitly derives from an ABC ArrayNamespace?

A nominal approach like that will only work if you're using array-api-compat. And in an ideal scenario, there wouldn't be any need for array-api-compat. So for a complete solution, I think that structural typing (i.e. "static duck typing" with Protocols) is the way to go here.

[NeilGirdhar]

And in an ideal scenario, there wouldn't be any need for array-api-compat

Is this even realistic for any backend? Aren't we patching all of them?

So for a complete solution, I think that structural typing (i.e. "static duck typing" with Protocols) is the way to go here.

TBH, I'm not even convinced of this since the Array API will have different versions, and each backend may have itself a different version. Therefore, you're going to have a very hard designing a protocol that matches every backend you happened to import.

On the other hand, if you had a particular version of array-api-compat, you can have it generate the appropriate shims to each of your backends so that they all match a given version of the Array API.

[jorenham]

Is this even realistic for any backend? Aren't we patching all of them?

I might be wrong here, but I think that NumPy 2.3.0 is pretty close to fully supporting v2024.

But I don't think that requiring array-api-compat as a "middle-man" here is a good idea. The relevant libraries generally speaking care about performance. And I can imagine that wrapping call to array-api functionality using array-api-compat can have a pretty big impact.

TBH, I'm not even convinced of this since the Array API will have different versions, and each backend may have itself a different version. Therefore, you're going to have a very hard designing a protocol that matches every backend you happened to import.

The plan is to publish different array-api-typing releases that target individual array-api versions. We'll start with 2021, and then work our way up. We'll use a versioning scheme that includes the array-api version, and our own patch version number, in case of bugfix backports and such.

On the other hand, if you had a particular version of array-api-compat, you can have it generate the appropriate shims to each of your backends so that they all match a given version of the Array API.

I think we'll be able to get this all working without code generation :)
In jorenham/optype#25 I wrote down how I think we'll can do that on the static typing side. But I want to condense it into something that will resembles a plan for array-api-typing, once I have the bandwidth.

[NeilGirdhar]

But I don't think that requiring array-api-compat as a "middle-man" here is a good idea. The relevant libraries generally speaking care about performance. And I can imagine that wrapping call to array-api functionality using array-api-compat can have a pretty big impact.

Can you explain in which way you can prevent compat from being a middleman? Isn't that the way in which you get array namespaces? If people want to use numpy or jax.numpy, they're still free to do that and they'll have the annotations from those libraries. If people use array_namespace(...), then they'll get xpt.ArrayNamespace, and they'll get those annotations, right?

The plan is to publish different array-api-typing releases that target individual array-api versions. We'll start with 2021, and then work our way up. We'll use a versioning scheme that includes the array-api version, and our own patch version number, in case of bugfix backports and such.

For any version of array-api-typing, is it not that case that:

• an old version of jax.numpy may not match the modern protocol (for example, due to an added method), and
• a newer version of jax.numpy may not match an older protocol (for example, due to a changed signature)

Therefore, even if you have clearly versioned xpt, the user still has to match it to numpy and jax versions (or whatever library they're using). So the user has to do that manually in pyproject?

If the user gets it wrong, the protocols won't match and there may be type errors, right?

I think we'll be able to get this all working without code generation :)

Oh that would be great. Looking forward to seeing it!

[jorenham]

Can you explain in which way you can prevent compat from being a middleman?

I can, but that's a very long answer, as that would basically require me to explain the entire plan I have in mind (and implementing it is probably faster), but I plan to do this soon (see #20 (comment)). So I'll leave answering this untill then, if you don't mind.

Isn't that the way in which you get array namespaces? If people want to use numpy or jax.numpy, they're still free to do that and they'll have the annotations from those libraries. If people use array_namespace(...), then they'll get xpt.ArrayNamespace, and they'll get those annotations, right?

The array_namespace(a) function is indeed a way of accessing the array-api functionality. Another way would be to use a.__array_namespace__() directly, which would give you the same result. So the library that provides this namespace, could annotate the __array_namespace__() return type using array-api-typing, without having to use aray-api-compat. If they don't then the user could use array-api-typing themselves to e.g. typing.cast this unknown return type to the appropriate structural type provided by array-api-typing. In both cases, there is no need for array-api-compat.

(Oh haha this accidentally also answers your first question, so it wasn't as complicated as I thought)

For any version of array-api-typing, is it not that case that:

• an old version of jax.numpy may not match the modern protocol (for example, due to an added method), and
• a newer version of jax.numpy may not match an older protocol (for example, due to a changed signature)

Therefore, even if you have clearly versioned xpt, the user still has to match it to numpy and jax versions (or whatever library they're using). So the user has to do that manually in pyproject?

If the user gets it wrong, the protocols won't match and there may be type errors, right?

As far as I know, the array API versions are backwards compatible with each other. So if you use different libraries with different array-api versions, then the lowest one should be used to pin array-api-typing.