2_creating_table_one.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tableone.ipynb",
      "version": "0.3.2",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/tompollard/buenosaires2018/blob/master/tableone.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "TzGImuDOLiv2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Demonstrating the `tableone` package"
      ]
    },
    {
      "metadata": {
        "id": "9eiVuYlqLiv4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In research papers, it is common for the first table (\"Table 1\") to display summary statistics of the study data. The `tableone` package is used to create this table. For an introduction to basic statistical reporting in biomedical journals, we recommend reading the [SAMPL Guidelines](http://www.equator-network.org/wp-content/uploads/2013/07/SAMPL-Guidelines-6-27-13.pdf). For more reading on accurate reporting in health research, visit the [EQUATOR Network](http://www.equator-network.org/)."
      ]
    },
    {
      "metadata": {
        "id": "fy-forYjLiv6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## A note for users of `tableone`"
      ]
    },
    {
      "metadata": {
        "id": "FmxjR9gLLiv6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "While we have tried to use best practices in creating this package, automation of even basic statistical tasks can be unsound if done without supervision. We encourage use of `tableone` alongside other methods of descriptive statistics and, in particular, visualization to ensure appropriate data handling. \n",
        "\n",
        "It is beyond the scope of our documentation to provide detailed guidance on summary statistics, but as a primer we provide some considerations for choosing parameters when creating a summary table at: http://tableone.readthedocs.io/en/latest/bestpractice.html. \n",
        "\n",
        "*Guidance should be sought from a statistician when using `tableone` for a research study, especially prior to submitting the study for publication*."
      ]
    },
    {
      "metadata": {
        "id": "uXyt9-zeLiv7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Installation\n",
        "\n",
        "To install the package with pip, run the following command in your terminal: ``pip install tableone``. To install the package with Conda, run: ``conda install -c conda-forge tableone``. For more detailed installation instructions, refer to the [documentation](http://tableone.readthedocs.io/en/latest/install.html). To install in Colaboratory, use `!pip install tableone`"
      ]
    },
    {
      "metadata": {
        "id": "IjWaSBFALpu0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# install the tableone package\n",
        "!pip install tableone"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sMy68bSpLiv7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Importing libraries\n",
        "\n",
        "Before using the `tableone` package, we need to import it. We will also import `pandas` for loading our sample dataset and `matplotlib` for creating plots."
      ]
    },
    {
      "metadata": {
        "id": "L0tZayvFLiv8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# import libraries\n",
        "from tableone import TableOne\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a7RZmk93Liv_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loading sample data\n",
        "\n",
        "We begin by loading the data that we would like to summarize into a Pandas DataFrame. \n",
        "- Variables are in columns\n",
        "- Encounters/observations are in rows."
      ]
    },
    {
      "metadata": {
        "id": "8r6xySgFLiwA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# load sample data into a pandas dataframe\n",
        "url=\"https://raw.githubusercontent.com/tompollard/tableone/master/data/pn2012_demo.csv\"\n",
        "data=pd.read_csv(url)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gtZ47zn_LiwC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gO2B6usALiwH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Example 1: Simple summary of data with Table 1\n",
        "\n",
        "In this example we provide summary statistics across all of the data."
      ]
    },
    {
      "metadata": {
        "id": "tULY2D-lLiwH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# view the tableone docstring\n",
        "TableOne??"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SxmtlDTFLiwK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# create an instance of TableOne with the input arguments\n",
        "# firstly, with no grouping variable\n",
        "overall_table = TableOne(data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1sEUWpGtLiwM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# view first 10 rows of tableone\n",
        "overall_table"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "h1ZWEVWaLiwQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Summary of the table**:\n",
        "- the first row ('`n`') displays a count of the encounters/observations in the input data.\n",
        "- the '`isnull`' column displays a count of the null values for the particular variable.\n",
        "- if categorical variables are not defined in the arguments, they are detected automatically.\n",
        "- continuous variables (e.g. '`age`') are summarized by '`mean (std)`'.\n",
        "- categorical variables (e.g. '`ascites`') are summarized by '`n (% of non-null values)`'."
      ]
    },
    {
      "metadata": {
        "id": "fm92S8pJLiwR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Exploring the warning raised by Hartigan's Dip Test\n",
        "\n",
        "Hartigan's Dip Test is a test for multimodality. The test has suggested that the `Age`, `SysABP`, and `Height` distributions may be multimodal. We'll plot the distributions here."
      ]
    },
    {
      "metadata": {
        "id": "uU6BWudKLiwR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data[['Age','SysABP','Height']].dropna().plot.kde(figsize=[12,8])\n",
        "plt.legend(['Age (years)', 'SysABP (mmHg)', 'Height (cm)'])\n",
        "plt.xlim([-30,250])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nnVOBmfNLiwU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Exploring the warning raised by Tukey's rule\n",
        "\n",
        "Tukey's rule has found far outliers in Height, so we'll look at this in a boxplot"
      ]
    },
    {
      "metadata": {
        "id": "0rs_4aYaLiwV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data[['Age','Height','SysABP']].boxplot(whis=3)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jsoHdD0CLiwX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In both cases it seems that there are values that may need to be taken into account when calculating the summary statistics. For `SysABP`, a clearly bimodal distribution, the researcher will need to decide how to handle the peak at ~0, perhaps by cleaning the data and/or describing the issue in the summary table. For `Height`, the researcher may choose to report median, rather than mean."
      ]
    },
    {
      "metadata": {
        "id": "7GymPeouLiwY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Example 2: Table 1 without stratification\n",
        "\n",
        "In this example we provide summary statistics across all of the data, specifying columns, categorical variables, and non-normal variables."
      ]
    },
    {
      "metadata": {
        "id": "bUmieBVeLiwZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# columns to be summarized\n",
        "columns = ['Age', 'SysABP', 'Height', 'Weight', 'ICU', 'death']\n",
        "\n",
        "# columns containing categorical variables\n",
        "categorical = ['ICU', 'death']\n",
        "\n",
        "# non-normal variables\n",
        "nonnormal = ['Age']\n",
        "\n",
        "# alternative labels\n",
        "labels={'LOS': 'Length of stay', 'death': 'mortality'}\n",
        "\n",
        "# create tableone with the input arguments\n",
        "mytable = TableOne(data, columns=columns, categorical=categorical, \n",
        "nonnormal=nonnormal, labels=labels)\n",
        "mytable"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2KlwiSEVLiwe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Summary of the table**:\n",
        "\n",
        "- as before, except that the variables are explicitly defined in the input arguments.\n",
        "- continuous variables are now summarized by '`median [IQR]`' if specified as `nonnormal`.\n",
        "- the labels argument means that 'sex' is now shown as 'gender, and 'trt' is now 'treatment'."
      ]
    },
    {
      "metadata": {
        "id": "W1NqXLorLiwe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Example 3: Table 1 with stratification\n",
        "\n",
        "In this example, we group data across a categorical variable."
      ]
    },
    {
      "metadata": {
        "id": "YUpDbu_YLiwg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# optionally, a categorical variable for stratification\n",
        "groupby = ['death']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PJRIQObRLiwi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# create an instance of TableOne with the input arguments\n",
        "grouped_table = TableOne(data, columns, categorical, groupby, nonnormal)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pOtp19i9Liwk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# view first 10 rows of tableone\n",
        "grouped_table"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OZRKoAYaLiwm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Summary of the table**:\n",
        "- data is now summarized across the groups specified in the `groupby` argument.\n",
        "- as before, the summary statistics are either '`mean (std)`', '`median [IQR]`', or '`n (% of non-null values)`'."
      ]
    },
    {
      "metadata": {
        "id": "1TJqkgAkLiwo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Computing p values\n",
        "\n",
        "We can run a test to compute p values by setting the ``pval`` argument to True."
      ]
    },
    {
      "metadata": {
        "id": "a9LGcvJeLiwo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# create grouped_table with p values\n",
        "grouped_table = TableOne(data, columns, categorical, groupby, nonnormal, pval = True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XQpZNMryLiwt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# view first 10 rows of tableone\n",
        "grouped_table"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YcSctv-WLiwx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Summary of the table**:\n",
        "- the '`ptest`' column displays the name of the test used to compare the groups.\n",
        "- the '`pval`' column displays the p value generated by the test in the '`ptest`' column, to 3 decimal places."
      ]
    },
    {
      "metadata": {
        "id": "4GmFofI0Liwy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Exporting the table to file (LaTeX, CSV, etc)\n",
        "\n",
        "Tables can be exported to file in various formats, including:\n",
        "- LaTeX\n",
        "- CSV\n",
        "- HTML\n",
        "\n",
        "To export the table, call the relevant `to_<format>()` method on the DataFrame."
      ]
    },
    {
      "metadata": {
        "id": "KTM6-t_GLiwz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Save table to LaTeX\n",
        "fn = 'tableone.tex'\n",
        "grouped_table.to_latex(fn)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GTSS9fBELiw1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Save table to HTML\n",
        "fn2 = 'tableone.html'\n",
        "grouped_table.to_html(fn2)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}