pandas3 DataFrames with text Series (pyarrow)

[PhilipYip1988]

Currently a Series in a DataFrame that has text has the dtype object:

import numpy as np
import pandas as pd
# pd.options.future.infer_string = True

df = pd.DataFrame({'text1': ['a', 'b', 'c', 'd', 'e'],
                   'text2': ['aa', 'bb', 'cc', 'dd', 'ee']})

And Spyder opens them as expected:

Using an dtype of object is problematic as the Series can contain str and numeric values and a TypeError occurs essentially with operations like '2'+ 2. In the future (pandas 3) Series in DataFrames with text will default to the dtype string [pyarrow_numpy]:

import numpy as np
import pandas as pd
pd.options.future.infer_string = True

df = pd.DataFrame({'text1': ['a', 'b', 'c', 'd', 'e'],
                   'text2': ['aa', 'bb', 'cc', 'dd', 'ee']})

Unfortunately when this option is enabled in Spyder 6.0.4 (standalone) with a Miniforge conda environment:

conda create -n spyder-env spyder-kernels python seaborn scikit-learn pyarrow sympy openpyxl xlrd xlsxwriter lxml sqlalchemy tabulate pyqt ffmpeg ruff

And the DataFrame is opened, there is an error:

Spyder was unable to retrieve the value of this variable from the console.

The error message was:

The 'pyarrow' package is required to open this variable. Unfortunately its not part of our installer, which means your variable can't be displayed by Spyder.

Note: Please don't report this problem on GitHub, there's nothing to do about it.

Although it says not to report, given this is going to be the future default behaviour of every Series with text. It is probably worthwhile adding pyarrow to the standalone installer and displaying the DataFrame (or Series or Index) in the same way as a DataFrame (or Series or Index) with dtype of object displays. Otherwise this error message will frequently occur when pandas version 3 is released.

[jitseniesen]

Thanks for alerting us to this. I did read about this before but I forgot about it. I agree that we do need to do something.

My worry is that I seem to remember that pyarrow is a big dependency. We do have plans to circumvent the requirement that packages like are installed in the runtime environment to be able to view variables in Spyder, but that may come too late. Do you have any idea of when pandas is planning to release version 3? It seems the current plan is to have a version 2.3 first.

[PhilipYip1988]

I'm not sure when pandas 3 is due out, I remember reading an article a while ago which mentioned September 2024 but only 2.2.3 was released then. The 3.0 milestone is 97 % complete.

I guess use of pyarrow is the most significant change and its going to be an option in a version 2.x probably until its stable enough for it to be a default in a 3.0 release.