display spark datasets and dataframes with table widget

[scottdraves]

see @jpallas comments #6993

indeed. the widget should show just what spark normally shows ascii-formatted tables (sometimes very wide).

is there a way to configure show method to use our repr?
maybe via https://github.com/jupyter/jvm-repr?
add a new method or a subclass?

after the tables support streaming #7006
then we can connect them directly to that.

[scottdraves]

related: #7041

[jpallas]

show is defined directly by Dataset and has no return value, so I don't think there's any way to get it into a table short of scraping the output (or doing unspeakable things with cglib).

[scottdraves]

yea should be a display handler, installed by jvm-repr.
maybe start with 1000 rows.

[scottdraves]

@jpallas on the other issue you said:

(I've prototyped this).

care to share? jarek could pick up your work if you think it's on the right track.

[jpallas]

What I did is just a few lines:

implicit class DatasetOps(ds: org.apache.spark.sql.Dataset[_]) {
    def display(rows: Int = 20) = {
        // I do not understand why this import is necessary
        import com.twosigma.beakerx.scala.table.TableDisplay
        val columns = ds.columns
        val rowVals = ds.toDF.take(rows)
        val t = new TableDisplay(rowVals map (row => (columns zip row.toSeq).toMap))
        t.display()
    }
}

(There is something strange going on with imports and visibility in the Scala kernel, maybe some odd interaction with the way the interpreter wraps things.)

Example with a Dataset[Fields] where case class Fields(first: String, second: String, third: Int):