refactor: #718 only drop TimestampSeries #1274

cmp0xff · 2025-07-13T06:40:48Z

Addresses CLEAN: Investigate whether TimestampSeries, TimedeltaSeries, etc. can be removed #718
Tests added: Please use assert_type() to assert the type of any return value

tests/test_timefuncs.py

pandas-stubs/core/series.pyi

tests/test_series.py

tests/test_frame.py

tests/test_scalars.py

Dr-Irv · 2025-07-23T16:53:15Z

@cmp0xff you have a number of PRs submitted while I was out on vacation for 2 weeks. Can you let me know which ones I should prioritize for review?

cmp0xff · 2025-07-23T18:15:12Z

Hi @Dr-Irv, I hope you had a nice vacation. My pull requests are categorised below. Each category is independent, but those in a higher position have a slightly higher priority in my opinion.

`Series`: arithmetic operations

The following two PRs are independent. They migrate test_series.py to a subfolder series, and add quite a few test_*.py files there.

add: feat(series): #1098 arithmetic addition #1275
truediv: feat(series): #1098 arithmetic truediv #1280

`DataFrame.to_dict`

fix(DataFrame): #799 to_dict #1283

`Index.append`

feat(index): append #1282

`Series`: address #718

refactor: #718 only drop TimestampSeries #1274 - this is a prerequisite for the next one.
refactor: #718 also drop TimedeltaSeries #1273

Dr-Irv

Thanks for doing this. It's a lot of good work.

Main thing - if I'm going to merge this PR, it needs to be in a state where we don't need the followup PR.

Basic rule - we don't put ignore in the tests unless we are testing that the stubs should not accept something that is invalid. You have places where you have added ignore in the tests and I won't merge that in (unless we know it is a bug in the type checker)

docs/philosophy.md

Dr-Irv · 2025-07-24T20:59:41Z

tests/test_frame.py

+    check(assert_type(s + summer, pd.Series), pd.Series)  # type: ignore[assert-type]
+    check(assert_type(s + df["y"], pd.Series), pd.Series)  # type: ignore[assert-type]
+    check(assert_type(summer + summer, pd.Series), pd.Series)  # type: ignore[assert-type]


don't want to have ignore in the tests. Fix the types to make this work.

These are not there anymore.

tests/test_frame.py

tests/test_scalars.py

tests/test_timefuncs.py

pandas-stubs/core/series.pyi

Dr-Irv · 2025-07-24T21:28:34Z

Hi @Dr-Irv, I hope you had a nice vacation. My pull requests are categorised below. Each category is independent, but those in a higher position have a slightly higher priority in my opinion.

I've reviewed them all, except #1273 as noted there.

Thanks for all the great work.

cmp0xff · 2025-07-24T22:45:42Z

I've reviewed them all, except #1273 as noted there.

Thanks for all the great work.

Thank you very much for your quick and thorough reviews. I will be able to work on them next week.

…les#r2229555145

…74/files#r2229550572

…les#r2229581983

Dr-Irv

This is pretty close. There are 2 main issues reflected in the comments.

I think the changes you made for __sub__() and possibly __mul__() and related methods to handle int, float, etc., should be a separate PR, similar to what you did for _add__() and __div__(). If we can get that working, and then do this one, it will be easier to make sure all the tests are still working right.
I don't want to merge with any new ignore in the tests, because if someone pulls main, they will get something we know is broken. But I suggest a plan for handling that in the comments.

Thanks again for the great work on this.

tests/series/test_series.py

Dr-Irv · 2025-08-07T18:30:54Z

tests/series/test_series.py

+    # Will be fixed after removing TimedeltaSeries, see Series.__sub__ in series.pyi
+    check(assert_type(ss, pd.Series), pd.Series)  # type: ignore[assert-type]


I'm don't want to merge until we deal with this, because everything in main should pass all the tests. But here is what I'd like to do. Once I'm OK with this PR, I won't merge it, and then the TimedeltaSeries PR can merge into this branch where the line above should get fixed.

I could put this branch into the repo once approved so it becomes the new PR target of the next PR, and then once that is approved and merged, we do a new PR from that branch to main

Now removed

To be more precise, there are a few cases where the return type is Series[Timedelta], instead of the to-be-removed TimedeltaSeries. When I put any of them as TimedeltaSeries, ss cannot be properly recognised.

Dr-Irv · 2025-08-07T18:48:12Z

pandas-stubs/core/indexes/accessors.pyi

 # is invoked, but because of how Series.dt is hooked in and that we may not know the
 # type of the series, we don't know which kind of series was ...ed
 # in to the dt accessor

 _DTTimestampTimedeltaReturnType = TypeVar(
-    "_DTTimestampTimedeltaReturnType",
-    bound=Series | TimestampSeries | TimedeltaSeries | DatetimeIndex | TimedeltaIndex,
+    "_DTTimestampTimedeltaReturnType", bound=Series | DatetimeIndex | TimedeltaIndex


Not clear why TimedeltaSeries is removed at this point. Or shouldn't it be bound=Series | Series[Timestamp] | TimedeltaSeries | DatetimeIndex | TimedeltaIndex ?

I know that Series includes the other 2, but I'd like to keep this as close as possible to what was there before.

Dr-Irv · 2025-08-07T18:58:37Z

pandas-stubs/core/series.pyi

+    @overload
+    def __sub__(self: Series[S1C], other: Series[Never]) -> Series: ...
+    @overload
    def __sub__(
-        self: Series[Timestamp],
-        other: Timedelta | TimedeltaSeries | TimedeltaIndex | np.timedelta64,
-    ) -> TimestampSeries: ...
+        self: Series[int], other: _T_COMPLEX | Sequence[_T_COMPLEX] | Series[_T_COMPLEX]
+    ) -> Series[_T_COMPLEX]: ...
+    @overload
+    def __sub__(self: Series[int], other: np_ndarray_anyint) -> Series[int]: ...
+    @overload
+    def __sub__(self: Series[int], other: np_ndarray_float) -> Series[float]: ...
+    @overload
+    def __sub__(self: Series[int], other: np_ndarray_complex) -> Series[complex]: ...
    @overload
    def __sub__(
-        self: Series[Timedelta],
-        other: Timedelta | TimedeltaSeries | TimedeltaIndex | np.timedelta64,
-    ) -> TimedeltaSeries: ...
+        self: Series[float],
+        other: int | Sequence[int] | np_ndarray_anyint | np_ndarray_float | Series[int],
+    ) -> Series[float]: ...
+    @overload
+    def __sub__(
+        self: Series[float],
+        other: _T_COMPLEX | Sequence[_T_COMPLEX] | Series[_T_COMPLEX],
+    ) -> Series[_T_COMPLEX]: ...
+    @overload


I think these changes for __sub__() and __rsub__() and sub() should be in a separate PR like you did for add and div, with the tests like you did there.

Good idea, we have merged #1311, #1312, #1314 and #1332.

pandas-stubs/core/series.pyi

…18-drop-tss

cmp0xff · 2025-08-20T20:50:49Z

Before making this PR as "ready for review", I probably can still clean up the sub family, make some simplifications and homogenise __sub__ and sub.

cmp0xff · 2025-08-20T20:53:09Z

tests/series/arithmetic/str/test_add.py

+    if sys.version_info >= (3, 11):
+        check(assert_type(r0 + left, "npt.NDArray[np.str_]"), pd.Series, str)
+    else:
+        check(assert_type(r0 + left, Any), pd.Series, str)


This is weird. On my local machine it does not pass. In GitHub CI it has to be like this in order to pass.

I was able to replicate the behavior locally with a python 3.10 and python 3.11 environment. I added this before the if statement:

reveal_type(r0 + left) reveal_type(r0) reveal_type(r0.__add__(left))

With pyright and python 3.10, I get this:

c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:54:17 - information: Type of "r0 + left" is "Any" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:55:17 - information: Type of "r0" is "ndarray[tuple[int, ...], dtype[str_]]" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:56:17 - information: Type of "r0.__add__(left)" is "Any"

With pyright and python 3.11, I get this:

c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:54:17 - information: Type of "r0 + left" is "ndarray[tuple[Any, ...], dtype[str_]]" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:55:17 - information: Type of "r0" is "ndarray[tuple[Any, ...], dtype[str_]]" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:56:17 - information: Type of "r0.__add__(left)" is "ndarray[tuple[Any, ...], dtype[str_]]"

Note the difference in the second reveal type. In python 3.10, it has for r0 ndarray[tuple[int,...], dtype[str_]] while for python 3.11 it has ndarray[tuple[Any,...,dtype[str_]]

Anyhow, can you put a comment above the if statement that indicates why you did this (you could link to this comment).

I think this is because python 3.10 is using numpy 2.2.6 (the last numpy release that supports python 3.10), while python 3.11 is using numpy 2.3.2 .

Once pandas 3.0 is released, we're going to drop python 3.10 support, and I'll do that here as well.

cmp0xff · 2025-08-21T14:48:42Z

tests/series/arithmetic/test_sub.py

+    check(assert_type(left_ts.rsub(s), pd.Series), pd.Series, pd.Timedelta)
+    check(assert_type(left_ts.rsub(a), pd.Series), pd.Series, pd.Timedelta)


datetime - Series[Any] can either be timedelta-like or datetime-like, depending on Any. I would not give an exact type here.

Yes, that makes sense.

Dr-Irv · 2025-08-21T13:45:45Z

attempt.py

why is this file in this PR?

Dr-Irv · 2025-08-21T13:48:52Z

pandas-stubs/core/indexes/accessors.pyi

+    ) -> TimestampProperties: ...
+    @overload
+    def __get__(
+        self, instance: Series[Timedelta], owner: Any


Use TimedeltaSeries here or TimedeltaSeries | Series[Timedelta] and then change in the PR that will remove TimedeltaSeries

Dr-Irv · 2025-08-21T13:51:11Z

tests/series/arithmetic/str/test_add.py

Can you put this test and the other tests related to Series[str], as well as any changes related to making these tests work, in a separate PR?

I'll comment on the test for now, but want to keep the PRs more focused.

Dr-Irv · 2025-08-21T13:52:05Z

tests/series/arithmetic/str/test_add.py

+    """Testpd.Series[str]+ Python native str"""
+    r0 = "right"
+


Can we add tests that check that things like left + 5 is caught by the type checker as invalid? (same here and in the other test funcs)

Dr-Irv · 2025-08-21T13:57:55Z

tests/series/arithmetic/test_sub.py

+    check(assert_type(s - left_ts, pd.Series), pd.Series, pd.Timedelta)

    check(assert_type(left_ts.sub(s), "TimedeltaSeries"), pd.Series, pd.Timedelta)

-    check(assert_type(left_ts.rsub(s), "TimedeltaSeries"), pd.Series, pd.Timedelta)
+    check(assert_type(left_ts.rsub(s), pd.Series), pd.Series, pd.Timedelta)


I'm OK with this change, but I think it would be worth having a test that is s - left_td where left_td is a series of Timedelta. That illustrates why we can't infer the subtype of the Series.

Dr-Irv · 2025-08-21T14:29:29Z

pandas-stubs/core/series.pyi

    def __mul__(
-        self, other: timedelta | Timedelta | TimedeltaSeries | np.timedelta64
+        self: Series[bool],
+        other: timedelta | np.timedelta64 | np_ndarray_td | TimedeltaSeries,
+    ) -> TimedeltaSeries: ...
+    @overload
+    def __mul__(self: Series[bool], other: Series[Timedelta]) -> Series[Timedelta]: ...  # type: ignore[overload-overlap]
+    @overload
+    def __mul__(
+        self: Series[int],
+        other: timedelta | np.timedelta64 | np_ndarray_td | TimedeltaSeries,
+    ) -> TimedeltaSeries: ...
+    @overload
+    def __mul__(self: Series[int], other: Series[Timedelta]) -> Series[Timedelta]: ...
+    @overload
+    def __mul__(
+        self: Series[float],
+        other: timedelta | np.timedelta64 | np_ndarray_td | TimedeltaSeries,
+    ) -> TimedeltaSeries: ...
+    @overload
+    def __mul__(self: Series[float], other: Series[Timedelta]) -> Series[Timedelta]: ...


Is it possible to combine these overloads like this?

@overload def __mul__( self: Series[bool] | Series[int] | Series[float], other: timedelta | np.timedelta64 | np_ndarray_td | TimedeltaSeries, ) -> TimedeltaSeries: ... @overload def __mul__(self: Series[bool] | Series[int] | Series[float],, other: Series[Timedelta]) -> Series[Timedelta]: ... # type: ignore[overload-overlap]

Dr-Irv · 2025-08-21T14:33:56Z

pandas-stubs/core/series.pyi

+            timedelta
+            | np.timedelta64
+            | np_ndarray_td
+            | TimedeltaIndex
+            | Series[Timedelta]
+            | TimedeltaSeries


you can add bool | int | float here (and in truediv()) and add appropriate tests. (maybe in the TimedeltaSeries removal PR)

Dr-Irv · 2025-08-21T14:36:46Z

pandas-stubs/core/series.pyi

    def median(
-        self,
+        self: Series[float],


Suggested change

self: Series[float],

self: Series[float] | Series[int]

And then for Series[bool], median() returns np.floating

Dr-Irv · 2025-08-21T14:40:30Z

pandas-stubs/core/series.pyi

+    @overload
+    def to_numpy(
+        self,
+        dtype: DTypeLike | None = None,
+        copy: bool = False,
+        na_value: Scalar = ...,
+        **kwargs,
+    ) -> np_1darray: ...


I don't think you need this overload because IndexOpsMixin has it. Then you can get rid of the ignores in _SeriesSubClassBase

Dr-Irv · 2025-08-21T15:58:21Z

tests/series/arithmetic/str/test_add.py

+    if sys.version_info >= (3, 11):
+        check(assert_type(r0 + left, "npt.NDArray[np.str_]"), pd.Series, str)
+    else:
+        check(assert_type(r0 + left, Any), pd.Series, str)


I was able to replicate the behavior locally with a python 3.10 and python 3.11 environment. I added this before the if statement:

reveal_type(r0 + left) reveal_type(r0) reveal_type(r0.__add__(left))

With pyright and python 3.10, I get this:

c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:54:17 - information: Type of "r0 + left" is "Any" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:55:17 - information: Type of "r0" is "ndarray[tuple[int, ...], dtype[str_]]" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:56:17 - information: Type of "r0.__add__(left)" is "Any"

With pyright and python 3.11, I get this:

c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:54:17 - information: Type of "r0 + left" is "ndarray[tuple[Any, ...], dtype[str_]]" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:55:17 - information: Type of "r0" is "ndarray[tuple[Any, ...], dtype[str_]]" c:\Code\pandas-stubs\tests\series\arithmetic\str\test_add.py:56:17 - information: Type of "r0.__add__(left)" is "ndarray[tuple[Any, ...], dtype[str_]]"

Note the difference in the second reveal type. In python 3.10, it has for r0 ndarray[tuple[int,...], dtype[str_]] while for python 3.11 it has ndarray[tuple[Any,...,dtype[str_]]

Anyhow, can you put a comment above the if statement that indicates why you did this (you could link to this comment).

I think this is because python 3.10 is using numpy 2.2.6 (the last numpy release that supports python 3.10), while python 3.11 is using numpy 2.3.2 .

Once pandas 3.0 is released, we're going to drop python 3.10 support, and I'll do that here as well.

cmp0xff mentioned this pull request Jul 13, 2025

refactor: #718 also drop TimedeltaSeries #1273

Draft

2 tasks

cmp0xff marked this pull request as ready for review July 13, 2025 07:05

cmp0xff changed the title ~~fix: #718 only drop TimestampSeries~~ refactor: #718 only drop TimestampSeries Jul 13, 2025