Add quantile to DataFrame and Series #318

V1NAY8 · 2020-11-06T11:24:56Z

Closes #315

Functionality w.r.t pandas for quantile is implemented i.e. df.agg['quantile',...] , df.quantile() , Series.quantile()
Added tests and documentation.

@sethmlarson Please Review 😄

P.S. Need to change commit message while merging to master from percentile to quantile 😃

elasticmachine · 2020-11-06T11:24:58Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

sethmlarson

Thanks for all these great changes!!! 🚀 This functionality will be very useful.
I left you a bunch of comments, I'm going to wait for this PR to land before reviewing the .groupby().quantile() PR.

eland/dataframe.py

eland/operations.py

eland/query_compiler.py

eland/series.py

eland/tests/dataframe/test_metrics_pytest.py

eland/tests/operations/test_map_pd_aggs_to_es_aggs_pytest.py

eland/tests/series/test_metrics_pytest.py

sethmlarson · 2020-11-06T15:23:03Z

jenkins test this please

V1NAY8 · 2020-11-19T13:17:22Z

@sethmlarson Can we merge mode PR into master, because this will have conflicts in operations.py or can reuse some if statements if possible? 😃

V1NAY8 · 2021-01-07T19:37:19Z

@sethmlarson This PR is ready. Please Review 😄

V1NAY8 · 2021-04-20T09:33:33Z

@stevedodson Please review this and let me know if any changes are required. 😄

V1NAY8 · 2021-05-20T16:02:13Z

Hi, If this can be merged. I can progress on #316 and #319

stevedodson · 2021-05-20T16:10:46Z

@V1NAY8 - sorry for the delay, I will review with @sethmlarson next week. Many thanks for your help!

V1NAY8 · 2021-05-20T16:12:51Z

Sure, I am happy to contribute more. I am much used to the code base. Learning Elastic Search as well! Thanks.

stevedodson · 2021-05-26T08:19:43Z

jenkins test this please

stevedodson

The following code below shows an issue with pandas (we should raise as an issue if it doesn't exist already). Therefore, don't add bool column types in tests for now.

Python 3.9.5 (default, May 18 2021, 12:31:01) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.23.1 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.23.1
Python 3.9.5 (default, May 18 2021, 12:31:01) 
[Clang 10.0.0 ] on darwin
import pandas as pd
pd.__version__
Out[3]: '1.2.4'
df = pd.DataFrame({'b': [False, True]})
df
Out[5]: 
       b
0  False
1   True
df.dtypes
Out[6]: 
b    bool
dtype: object
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   b       2 non-null      bool 
dtypes: bool(1)
memory usage: 130.0 bytes
df
Out[8]: 
       b
0  False
1   True
df.quantile(q=0.5)
Traceback (most recent call last):
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-9-21ac846264fe>", line 1, in <module>
    df.quantile(q=0.5)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/frame.py", line 9266, in quantile
    result = data._mgr.quantile(
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 491, in quantile
    block = b.quantile(axis=axis, qs=qs, interpolation=interpolation)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 1592, in quantile
    result = nanpercentile(
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/nanops.py", line 1675, in nanpercentile
    return np.percentile(values, q, axis=axis, interpolation=interpolation)
  File "<__array_function__ internals>", line 5, in percentile
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3818, in percentile
    return _quantile_unchecked(
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3937, in _quantile_unchecked
    r, k = _ureduce(a, func=_quantile_ureduce_func, q=q, axis=axis, out=out,
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3515, in _ureduce
    r = func(a, **kwargs)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 4064, in _quantile_ureduce_func
    r = _lerp(x_below, x_above, weights_above, out=out)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3961, in _lerp
    diff_b_a = subtract(b, a)
TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
df.quantile(q=0.5, numeric_only=True)
Traceback (most recent call last):
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-10-70bdf370f901>", line 1, in <module>
    df.quantile(q=0.5, numeric_only=True)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/frame.py", line 9266, in quantile
    result = data._mgr.quantile(
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 491, in quantile
    block = b.quantile(axis=axis, qs=qs, interpolation=interpolation)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 1592, in quantile
    result = nanpercentile(
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/pandas/core/nanops.py", line 1675, in nanpercentile
    return np.percentile(values, q, axis=axis, interpolation=interpolation)
  File "<__array_function__ internals>", line 5, in percentile
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3818, in percentile
    return _quantile_unchecked(
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3937, in _quantile_unchecked
    r, k = _ureduce(a, func=_quantile_ureduce_func, q=q, axis=axis, out=out,
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3515, in _ureduce
    r = func(a, **kwargs)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 4064, in _quantile_ureduce_func
    r = _lerp(x_below, x_above, weights_above, out=out)
  File "/Users/steve/anaconda3/envs/eland_master/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3961, in _lerp
    diff_b_a = subtract(b, a)
TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

eland/dataframe.py

tests/dataframe/test_metrics_pytest.py

tests/series/test_metrics_pytest.py

V1NAY8 · 2021-05-31T14:37:37Z

@stevedodson Apologies for the delay :(
Ask jenkins to have fun

sethmlarson · 2021-05-31T18:21:08Z

Jenkins test this please

V1NAY8 · 2021-05-31T19:02:27Z

CI is good

sethmlarson

This PR is great, love the test suite so far. Handful of comments and questions

eland/operations.py

stevedodson · 2021-06-08T09:32:06Z

Jenkins test this please

sethmlarson

LGTM, thanks!!!

V1NAY8 · 2021-06-08T18:05:42Z

Thank You ❤️

sethmlarson suggested changes Nov 6, 2020

View reviewed changes

V1NAY8 requested a review from sethmlarson November 7, 2020 15:51

V1NAY8 force-pushed the add-percentile branch from ed9c89a to 22e5133 Compare January 7, 2021 19:29

stevedodson suggested changes May 26, 2021

View reviewed changes

eland/dataframe.py Show resolved Hide resolved

tests/dataframe/test_metrics_pytest.py Outdated Show resolved Hide resolved

tests/series/test_metrics_pytest.py Outdated Show resolved Hide resolved

V1NAY8 added 6 commits May 31, 2021 20:02

Add percentile to DataFrame and Series

c00fd36

Requested changes

96c44d1

Requested changes

38c3d5b

Rebase changes

e3ad601

Fix typo

13f409a

Fix Pandas quantile incompatibility with boolean

b1297eb

V1NAY8 force-pushed the add-percentile branch from 29e3b3b to b1297eb Compare May 31, 2021 14:33

V1NAY8 requested a review from stevedodson May 31, 2021 14:37

sethmlarson suggested changes Jun 2, 2021

View reviewed changes

Requested changes

e46b438

V1NAY8 requested a review from sethmlarson June 3, 2021 19:17

sethmlarson approved these changes Jun 8, 2021

View reviewed changes

sethmlarson merged commit e9c0b89 into elastic:master Jun 8, 2021

V1NAY8 deleted the add-percentile branch June 8, 2021 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add quantile to DataFrame and Series #318

Add quantile to DataFrame and Series #318

V1NAY8 commented Nov 6, 2020

elasticmachine commented Nov 6, 2020

sethmlarson left a comment •

edited

Loading

sethmlarson commented Nov 6, 2020

V1NAY8 commented Nov 19, 2020

V1NAY8 commented Jan 7, 2021

V1NAY8 commented Apr 20, 2021

V1NAY8 commented May 20, 2021

stevedodson commented May 20, 2021

V1NAY8 commented May 20, 2021

stevedodson commented May 26, 2021

stevedodson left a comment

V1NAY8 commented May 31, 2021 •

edited

Loading

sethmlarson commented May 31, 2021

V1NAY8 commented May 31, 2021

sethmlarson left a comment

stevedodson commented Jun 8, 2021

sethmlarson left a comment

V1NAY8 commented Jun 8, 2021

Add quantile to DataFrame and Series #318

Add quantile to DataFrame and Series #318

Conversation

V1NAY8 commented Nov 6, 2020

elasticmachine commented Nov 6, 2020

sethmlarson left a comment • edited Loading

Choose a reason for hiding this comment

sethmlarson commented Nov 6, 2020

V1NAY8 commented Nov 19, 2020

V1NAY8 commented Jan 7, 2021

V1NAY8 commented Apr 20, 2021

V1NAY8 commented May 20, 2021

stevedodson commented May 20, 2021

V1NAY8 commented May 20, 2021

stevedodson commented May 26, 2021

stevedodson left a comment

Choose a reason for hiding this comment

V1NAY8 commented May 31, 2021 • edited Loading

sethmlarson commented May 31, 2021

V1NAY8 commented May 31, 2021

sethmlarson left a comment

Choose a reason for hiding this comment

stevedodson commented Jun 8, 2021

sethmlarson left a comment

Choose a reason for hiding this comment

V1NAY8 commented Jun 8, 2021

sethmlarson left a comment •

edited

Loading

V1NAY8 commented May 31, 2021 •

edited

Loading