#Installing pycaret
!pip install pycaret
Collecting pycaret Downloading https://files.pythonhosted.org/packages/91/ae/000d825af8f7d9ff86808600f220e7ad57a873987fd6119c87dc4c5b1d91/pycaret-2.0-py3-none-any.whl (255kB) |████████████████████████████████| 256kB 4.9MB/s Requirement already satisfied: xgboost>=0.90 in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.90) Collecting pandas-profiling>=2.3.0 Downloading https://files.pythonhosted.org/packages/b9/94/ef8ef4517540d13406fcc0b8adfd75336e014242c69bd4162ab46931f36a/pandas_profiling-2.8.0-py2.py3-none-any.whl (259kB) |████████████████████████████████| 266kB 16.6MB/s Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.16.0) Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from pycaret) (3.2.5) Collecting scikit-learn>=0.23 Downloading https://files.pythonhosted.org/packages/5c/a1/273def87037a7fb010512bbc5901c31cfddfca8080bc63b42b26e3cc55b3/scikit_learn-0.23.2-cp36-cp36m-manylinux1_x86_64.whl (6.8MB) |████████████████████████████████| 6.8MB 19.6MB/s Collecting yellowbrick>=1.0.1 Downloading https://files.pythonhosted.org/packages/13/95/a14e4fdfb8b1c8753bbe74a626e910a98219ef9c87c6763585bbd30d84cf/yellowbrick-1.1-py3-none-any.whl (263kB) |████████████████████████████████| 266kB 43.5MB/s Requirement already satisfied: ipywidgets in /usr/local/lib/python3.6/dist-packages (from pycaret) (7.5.1) Requirement already satisfied: IPython in /usr/local/lib/python3.6/dist-packages (from pycaret) (5.5.0) Collecting pyLDAvis Downloading https://files.pythonhosted.org/packages/a5/3a/af82e070a8a96e13217c8f362f9a73e82d61ac8fff3a2561946a97f96266/pyLDAvis-2.1.2.tar.gz (1.6MB) |████████████████████████████████| 1.6MB 54.2MB/s Collecting mlflow Downloading https://files.pythonhosted.org/packages/00/2f/2529268d85af0a1521b0b7c137b63b731dff4784e1322fb3055403a959fb/mlflow-1.10.0-py3-none-any.whl (12.4MB) |████████████████████████████████| 12.4MB 243kB/s Collecting lightgbm>=2.3.1 Downloading https://files.pythonhosted.org/packages/0b/9d/ddcb2f43aca194987f1a99e27edf41cf9bc39ea750c3371c2a62698c509a/lightgbm-2.3.1-py2.py3-none-manylinux1_x86_64.whl (1.2MB) |████████████████████████████████| 1.2MB 53.9MB/s Requirement already satisfied: spacy in /usr/local/lib/python3.6/dist-packages (from pycaret) (2.2.4) Requirement already satisfied: wordcloud in /usr/local/lib/python3.6/dist-packages (from pycaret) (1.5.0) Collecting kmodes>=0.10.1 Downloading https://files.pythonhosted.org/packages/b2/55/d8ec1ae1f7e1e202a8a4184c6852a3ee993b202b0459672c699d0ac18fc8/kmodes-0.10.2-py2.py3-none-any.whl Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.4.3) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.6/dist-packages (from pycaret) (1.18.5) Collecting pyod Downloading https://files.pythonhosted.org/packages/77/4e/5767edaccbfc227914ca774cb6ca9b628a08cbb59b9b4839296953a63d34/pyod-0.8.1.tar.gz (93kB) |████████████████████████████████| 102kB 13.9MB/s Requirement already satisfied: umap-learn in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.4.6) Collecting DateTime>=4.3 Downloading https://files.pythonhosted.org/packages/73/22/a5297f3a1f92468cc737f8ce7ba6e5f245fcfafeae810ba37bd1039ea01c/DateTime-4.3-py2.py3-none-any.whl (60kB) |████████████████████████████████| 61kB 10.3MB/s Collecting catboost Downloading https://files.pythonhosted.org/packages/96/6c/6608210b29649267de52001b09e369777ee2a5cfe1c71fa75eba82a4f2dc/catboost-0.24-cp36-none-manylinux1_x86_64.whl (65.9MB) |████████████████████████████████| 65.9MB 71kB/s Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from pycaret) (3.2.2) Requirement already satisfied: cufflinks>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.17.3) Collecting datefinder>=0.7.0 Downloading https://files.pythonhosted.org/packages/0c/4f/29524c9ca35d2ba1a8a3c6c895b90fc92525cf0fe357f747133890953ebe/datefinder-0.7.1-py2.py3-none-any.whl Requirement already satisfied: plotly>=4.4.1 in /usr/local/lib/python3.6/dist-packages (from pycaret) (4.4.1) Requirement already satisfied: mlxtend in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.14.0) Requirement already satisfied: textblob in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.15.3) Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from pycaret) (0.10.1) Requirement already satisfied: gensim in /usr/local/lib/python3.6/dist-packages (from pycaret) (3.6.0) Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from pycaret) (1.0.5) Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from xgboost>=0.90->pycaret) (1.4.1) Requirement already satisfied: astropy>=4.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling>=2.3.0->pycaret) (4.0.1.post1) Collecting visions[type_image_path]==0.4.4 Downloading https://files.pythonhosted.org/packages/4a/03/5a45d542257830cf1d9da2cdc1c0bc6f55a9212937b70fdd6d7031b46d6c/visions-0.4.4-py3-none-any.whl (59kB) |████████████████████████████████| 61kB 9.3MB/s Collecting confuse>=1.0.0 Downloading https://files.pythonhosted.org/packages/b5/6d/bedc0d1068bd244cee05843313cbec6cebb9f01f925538269bababc6d887/confuse-1.3.0-py2.py3-none-any.whl (64kB) |████████████████████████████████| 71kB 11.3MB/s Requirement already satisfied: missingno>=0.4.2 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling>=2.3.0->pycaret) (0.4.2) Collecting phik>=0.9.10 Downloading https://files.pythonhosted.org/packages/01/5a/7ef1c04ce62cd72f900c06298dc2385840550d5c653a0dbc19109a5477e6/phik-0.10.0-py3-none-any.whl (599kB) |████████████████████████████████| 604kB 37.2MB/s Collecting tqdm>=4.43.0 Downloading https://files.pythonhosted.org/packages/28/7e/281edb5bc3274dfb894d90f4dbacfceaca381c2435ec6187a2c6f329aed7/tqdm-4.48.2-py2.py3-none-any.whl (68kB) |████████████████████████████████| 71kB 11.8MB/s Requirement already satisfied: requests>=2.23.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling>=2.3.0->pycaret) (2.23.0) Collecting htmlmin>=0.1.12 Downloading https://files.pythonhosted.org/packages/b3/e7/fcd59e12169de19f0131ff2812077f964c6b960e7c09804d30a7bf2ab461/htmlmin-0.1.12.tar.gz Collecting tangled-up-in-unicode>=0.0.6 Downloading https://files.pythonhosted.org/packages/4a/e2/e588ab9298d4989ce7fdb2b97d18aac878d99dbdc379a4476a09d9271b68/tangled_up_in_unicode-0.0.6-py3-none-any.whl (3.1MB) |████████████████████████████████| 3.1MB 46.3MB/s Requirement already satisfied: jinja2>=2.11.1 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling>=2.3.0->pycaret) (2.11.2) Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk->pycaret) (1.15.0) Collecting threadpoolctl>=2.0.0 Downloading https://files.pythonhosted.org/packages/f7/12/ec3f2e203afa394a149911729357aa48affc59c20e2c1c8297a60f33f133/threadpoolctl-2.1.0-py3-none-any.whl Requirement already satisfied: cycler>=0.10.0 in /usr/local/lib/python3.6/dist-packages (from yellowbrick>=1.0.1->pycaret) (0.10.0) Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.6/dist-packages (from ipywidgets->pycaret) (4.10.1) Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.6/dist-packages (from ipywidgets->pycaret) (3.5.1) Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.6/dist-packages (from ipywidgets->pycaret) (4.3.3) Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.6/dist-packages (from ipywidgets->pycaret) (5.0.7) Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (2.1.3) Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (49.2.0) Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (1.0.18) Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (4.4.2) Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (0.8.1) Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (0.7.5) Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.6/dist-packages (from IPython->pycaret) (4.8.0) Requirement already satisfied: wheel>=0.23.0 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis->pycaret) (0.34.2) Requirement already satisfied: numexpr in /usr/local/lib/python3.6/dist-packages (from pyLDAvis->pycaret) (2.7.1) Requirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from pyLDAvis->pycaret) (3.6.4) Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyLDAvis->pycaret) (0.16.0) Collecting funcy Downloading https://files.pythonhosted.org/packages/ce/4b/6ffa76544e46614123de31574ad95758c421aae391a1764921b8a81e1eae/funcy-1.14.tar.gz (548kB) |████████████████████████████████| 552kB 52.3MB/s Collecting azure-storage-blob>=12.0 Downloading https://files.pythonhosted.org/packages/6d/3d/31614573e8a197db12d8ab47a7fd813f15bd4a4b5c64e85d23b865de5b9b/azure_storage_blob-12.3.2-py2.py3-none-any.whl (280kB) |████████████████████████████████| 286kB 46.6MB/s Collecting gorilla Downloading https://files.pythonhosted.org/packages/e3/56/5a683944cbfc77e429c6f03c636ca50504a785f60ffae91ddd7f5f7bb520/gorilla-0.3.0-py2.py3-none-any.whl Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (3.13) Collecting databricks-cli>=0.8.7 Downloading https://files.pythonhosted.org/packages/1e/57/5c2d6b83cb8753d12f548e89f91037632baa8289677c1b2ab2adf14bf6b2/databricks-cli-0.11.0.tar.gz (49kB) |████████████████████████████████| 51kB 8.3MB/s Collecting alembic Downloading https://files.pythonhosted.org/packages/60/1e/cabc75a189de0fbb2841d0975243e59bde8b7822bacbb95008ac6fe9ad47/alembic-1.4.2.tar.gz (1.1MB) |████████████████████████████████| 1.1MB 48.3MB/s Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Requirement already satisfied: sqlparse in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (0.3.1) Requirement already satisfied: Flask in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (1.1.2) Collecting querystring-parser Downloading https://files.pythonhosted.org/packages/4a/fa/f54f5662e0eababf0c49e92fd94bf178888562c0e7b677c8941bbbcd1bd6/querystring_parser-1.2.4.tar.gz Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (3.12.4) Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (7.1.2) Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (2.8.1) Collecting gitpython>=2.1.0 Downloading https://files.pythonhosted.org/packages/f9/1e/a45320cab182bf1c8656107b3d4c042e659742822fc6bff150d769a984dd/GitPython-3.1.7-py3-none-any.whl (158kB) |████████████████████████████████| 163kB 53.9MB/s Collecting prometheus-flask-exporter Downloading https://files.pythonhosted.org/packages/84/b0/7549899a75e7ef0be5f8969094d0709e5dcc51dc21d58cc86b8d69441dce/prometheus_flask_exporter-0.15.4.tar.gz Collecting docker>=4.0.0 Downloading https://files.pythonhosted.org/packages/3c/15/7a2f095a3b8b0fff9a0a5f56bd941e05fa958d4ca31105541001a5f7d3eb/docker-4.2.2-py2.py3-none-any.whl (144kB) |████████████████████████████████| 153kB 51.7MB/s Requirement already satisfied: cloudpickle in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (1.3.0) Collecting gunicorn; platform_system != "Windows" Downloading https://files.pythonhosted.org/packages/69/ca/926f7cd3a2014b16870086b2d0fdc84a9e49473c68a8dff8b57f7c156f43/gunicorn-20.0.4-py2.py3-none-any.whl (77kB) |████████████████████████████████| 81kB 11.8MB/s Requirement already satisfied: entrypoints in /usr/local/lib/python3.6/dist-packages (from mlflow->pycaret) (0.3) Collecting sqlalchemy<=1.3.13 Downloading https://files.pythonhosted.org/packages/af/47/35edeb0f86c0b44934c05d961c893e223ef27e79e1f53b5e6f14820ff553/SQLAlchemy-1.3.13.tar.gz (6.0MB) |████████████████████████████████| 6.0MB 49.7MB/s Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (1.1.3) Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (7.4.0) Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (1.0.2) Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (0.7.1) Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (3.0.2) Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (2.0.3) Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (1.0.2) Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (1.0.0) Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy->pycaret) (0.4.1) Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from wordcloud->pycaret) (7.0.0) Collecting combo Downloading https://files.pythonhosted.org/packages/0a/2a/61b6ac584e75d8df16dc27962aa5fe99d76b09da5b6710e83d4862c84001/combo-0.1.1.tar.gz Requirement already satisfied: numba>=0.35 in /usr/local/lib/python3.6/dist-packages (from pyod->pycaret) (0.48.0) Collecting suod Downloading https://files.pythonhosted.org/packages/a1/87/9170cabe1b5e10a7d095c0e28f2e30e7c1886a13f063de85d3cfacc06f4b/suod-0.0.4.tar.gz (2.1MB) |████████████████████████████████| 2.1MB 50.5MB/s Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from DateTime>=4.3->pycaret) (2018.9) Collecting zope.interface Downloading https://files.pythonhosted.org/packages/57/33/565274c28a11af60b7cfc0519d46bde4125fcd7d32ebc0a81b480d0e8da6/zope.interface-5.1.0-cp36-cp36m-manylinux2010_x86_64.whl (234kB) |████████████████████████████████| 235kB 58.6MB/s Requirement already satisfied: graphviz in /usr/local/lib/python3.6/dist-packages (from catboost->pycaret) (0.10.1) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pycaret) (2.4.7) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pycaret) (1.2.0) Requirement already satisfied: colorlover>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from cufflinks>=0.17.0->pycaret) (0.3.0) Requirement already satisfied: regex>=2017.02.08 in /usr/local/lib/python3.6/dist-packages (from datefinder>=0.7.0->pycaret) (2019.12.20) Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.6/dist-packages (from plotly>=4.4.1->pycaret) (1.3.3) Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim->pycaret) (2.1.0) Requirement already satisfied: networkx>=2.4 in /usr/local/lib/python3.6/dist-packages (from visions[type_image_path]==0.4.4->pandas-profiling>=2.3.0->pycaret) (2.4) Requirement already satisfied: attrs>=19.3.0 in /usr/local/lib/python3.6/dist-packages (from visions[type_image_path]==0.4.4->pandas-profiling>=2.3.0->pycaret) (19.3.0) Collecting imagehash; extra == "type_image_path" Downloading https://files.pythonhosted.org/packages/1a/5d/cc81830be3c4705a46cdbca74439b67f1017881383ba0127c41c4cecb7b3/ImageHash-4.1.0.tar.gz (291kB) |████████████████████████████████| 296kB 56.9MB/s Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling>=2.3.0->pycaret) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling>=2.3.0->pycaret) (3.0.4) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling>=2.3.0->pycaret) (1.24.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling>=2.3.0->pycaret) (2020.6.20) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from jinja2>=2.11.1->pandas-profiling>=2.3.0->pycaret) (1.1.1) Requirement already satisfied: jupyter-client in /usr/local/lib/python3.6/dist-packages (from ipykernel>=4.5.1->ipywidgets->pycaret) (5.3.5) Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.6/dist-packages (from ipykernel>=4.5.1->ipywidgets->pycaret) (5.1.1) Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.6/dist-packages (from widgetsnbextension~=3.5.0->ipywidgets->pycaret) (5.3.1) Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.3.1->ipywidgets->pycaret) (0.2.0) Requirement already satisfied: jupyter-core in /usr/local/lib/python3.6/dist-packages (from nbformat>=4.2.0->ipywidgets->pycaret) (4.6.3) Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.6/dist-packages (from nbformat>=4.2.0->ipywidgets->pycaret) (2.6.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->IPython->pycaret) (0.2.5) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.6/dist-packages (from pexpect; sys_platform != "win32"->IPython->pycaret) (0.6.0) Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis->pycaret) (1.9.0) Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis->pycaret) (1.4.0) Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis->pycaret) (0.7.1) Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis->pycaret) (8.4.0) Collecting azure-core<2.0.0,>=1.6.0 Downloading https://files.pythonhosted.org/packages/8b/00/efb68e2dda82139d732090fc3b7ff47fe6f34724ea7ba31e518a854b15c1/azure_core-1.7.0-py2.py3-none-any.whl (121kB) |████████████████████████████████| 122kB 52.7MB/s Collecting cryptography>=2.1.4 Downloading https://files.pythonhosted.org/packages/ba/91/84a29d6a27fd6dfc21f475704c4d2053d58ed7a4033c2b0ce1b4ca4d03d9/cryptography-3.0-cp35-abi3-manylinux2010_x86_64.whl (2.7MB) |████████████████████████████████| 2.7MB 56.0MB/s Collecting msrest>=0.6.10 Downloading https://files.pythonhosted.org/packages/d9/ed/8e1b75721ad983c1672cd968ad3ae374d2e94767edff6f0b72a15dfde933/msrest-0.6.18-py2.py3-none-any.whl (84kB) |████████████████████████████████| 92kB 13.8MB/s Requirement already satisfied: tabulate>=0.7.7 in /usr/local/lib/python3.6/dist-packages (from databricks-cli>=0.8.7->mlflow->pycaret) (0.8.7) Collecting python-editor>=0.3 Downloading https://files.pythonhosted.org/packages/c6/d3/201fc3abe391bbae6606e6f1d598c15d367033332bd54352b12f35513717/python_editor-1.0.4-py3-none-any.whl Collecting Mako Downloading https://files.pythonhosted.org/packages/a6/37/0e706200d22172eb8fa17d68a7ae22dec7631a0a92266634fb518a88a5b2/Mako-1.1.3-py2.py3-none-any.whl (75kB) |████████████████████████████████| 81kB 11.4MB/s Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from Flask->mlflow->pycaret) (1.1.0) Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from Flask->mlflow->pycaret) (1.0.1) Collecting gitdb<5,>=4.0.1 Downloading https://files.pythonhosted.org/packages/48/11/d1800bca0a3bae820b84b7d813ad1eff15a48a64caea9c823fc8c1b119e8/gitdb-4.0.5-py3-none-any.whl (63kB) |████████████████████████████████| 71kB 11.3MB/s Requirement already satisfied: prometheus_client in /usr/local/lib/python3.6/dist-packages (from prometheus-flask-exporter->mlflow->pycaret) (0.8.0) Collecting websocket-client>=0.32.0 Downloading https://files.pythonhosted.org/packages/4c/5f/f61b420143ed1c8dc69f9eaec5ff1ac36109d52c80de49d66e0c36c3dfdf/websocket_client-0.57.0-py2.py3-none-any.whl (200kB) |████████████████████████████████| 204kB 59.8MB/s Requirement already satisfied: importlib-metadata>=0.20; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy->pycaret) (1.7.0) Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.35->pyod->pycaret) (0.31.0) Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim->pycaret) (1.14.33) Requirement already satisfied: boto in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim->pycaret) (2.49.0) Requirement already satisfied: PyWavelets in /usr/local/lib/python3.6/dist-packages (from imagehash; extra == "type_image_path"->visions[type_image_path]==0.4.4->pandas-profiling>=2.3.0->pycaret) (1.1.1) Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.6/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets->pycaret) (19.0.2) Requirement already satisfied: nbconvert in /usr/local/lib/python3.6/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (5.6.1) Requirement already satisfied: Send2Trash in /usr/local/lib/python3.6/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (1.5.0) Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (0.8.3) Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.6/dist-packages (from cryptography>=2.1.4->azure-storage-blob>=12.0->mlflow->pycaret) (1.14.1) Collecting isodate>=0.6.0 Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB) |████████████████████████████████| 51kB 8.6MB/s Requirement already satisfied: requests-oauthlib>=0.5.0 in /usr/local/lib/python3.6/dist-packages (from msrest>=0.6.10->azure-storage-blob>=12.0->mlflow->pycaret) (1.3.0) Collecting smmap<4,>=3.0.1 Downloading https://files.pythonhosted.org/packages/b0/9a/4d409a6234eb940e6a78dfdfc66156e7522262f5f2fecca07dc55915952d/smmap-3.0.4-py2.py3-none-any.whl Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < "3.8"->catalogue<1.1.0,>=0.0.7->spacy->pycaret) (3.1.0) Requirement already satisfied: botocore<1.18.0,>=1.17.33 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim->pycaret) (1.17.33) Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim->pycaret) (0.10.0) Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim->pycaret) (0.3.3) Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (0.8.4) Requirement already satisfied: defusedxml in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (0.6.0) Requirement already satisfied: bleach in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (3.1.5) Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (1.4.2) Requirement already satisfied: testpath in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (0.4.4) Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.1.4->azure-storage-blob>=12.0->mlflow->pycaret) (2.20) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.10->azure-storage-blob>=12.0->mlflow->pycaret) (3.1.0) Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.18.0,>=1.17.33->boto3->smart-open>=1.2.1->gensim->pycaret) (0.15.2) Requirement already satisfied: webencodings in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (0.5.1) Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->pycaret) (20.4) Building wheels for collected packages: alembic Building wheel for alembic (PEP 517) ... done Created wheel for alembic: filename=alembic-1.4.2-cp36-none-any.whl size=159543 sha256=87afe1e447cc0f292505a2a15fe336265784a2ac8582dca0cda70514381189ec Stored in directory: /root/.cache/pip/wheels/1f/04/83/76023f7a4c14688c0b5c2682a96392cfdd3ee4449eaaa287ef Successfully built alembic Building wheels for collected packages: pyLDAvis, pyod, htmlmin, funcy, databricks-cli, querystring-parser, prometheus-flask-exporter, sqlalchemy, combo, suod, imagehash Building wheel for pyLDAvis (setup.py) ... done Created wheel for pyLDAvis: filename=pyLDAvis-2.1.2-py2.py3-none-any.whl size=97711 sha256=b1b90a9afd6e4366edef04af0d4f6e7596120607d71b0372b62c75be854a5fe0 Stored in directory: /root/.cache/pip/wheels/98/71/24/513a99e58bb6b8465bae4d2d5e9dba8f0bef8179e3051ac414 Building wheel for pyod (setup.py) ... done Created wheel for pyod: filename=pyod-0.8.1-cp36-none-any.whl size=105653 sha256=afae347204f267376e8bda921c829dbed085dfe4d76fbae06ecbf7aa6d9eba01 Stored in directory: /root/.cache/pip/wheels/2e/ca/18/727e9d98a41f5f4385a97d5b429f3a9c8fbee13f9780c18642 Building wheel for htmlmin (setup.py) ... done Created wheel for htmlmin: filename=htmlmin-0.1.12-cp36-none-any.whl size=27084 sha256=13ac8a83cd4b5cfbc39205fd7f3b52c17bcd5749291faf808b0a8db3879214de Stored in directory: /root/.cache/pip/wheels/43/07/ac/7c5a9d708d65247ac1f94066cf1db075540b85716c30255459 Building wheel for funcy (setup.py) ... done Created wheel for funcy: filename=funcy-1.14-py2.py3-none-any.whl size=32042 sha256=a7374bce770cbab81dc2daa997a4ec5acf4469a33e97b6a3a2349e5ebb013b7f Stored in directory: /root/.cache/pip/wheels/20/5a/d8/1d875df03deae6f178dfdf70238cca33f948ef8a6f5209f2eb Building wheel for databricks-cli (setup.py) ... done Created wheel for databricks-cli: filename=databricks_cli-0.11.0-cp36-none-any.whl size=90300 sha256=866f49cde1afee472821ae0e373186749fde9d84e65768697b395712d80c2036 Stored in directory: /root/.cache/pip/wheels/63/d0/4f/3deeca1f4c47a6aca7c2c6a6e2bf272391565dc86a7718a59b Building wheel for querystring-parser (setup.py) ... done Created wheel for querystring-parser: filename=querystring_parser-1.2.4-cp36-none-any.whl size=7079 sha256=59ea59c609685b1b972206b081d7d63af67ac5038b60e3cceed0a64ec3da0a30 Stored in directory: /root/.cache/pip/wheels/1e/41/34/23ebf5d1089a9aed847951e0ee375426eb4ad0a7079d88d41e Building wheel for prometheus-flask-exporter (setup.py) ... done Created wheel for prometheus-flask-exporter: filename=prometheus_flask_exporter-0.15.4-cp36-none-any.whl size=16454 sha256=b0623ae16407ce374879d5dacef00f80c7ca7196f5dc5287280a30f6ac2c65b7 Stored in directory: /root/.cache/pip/wheels/4f/b4/70/b18fa12c1c0a30fd542767dbbcdac225c6aae012fa1b3124e4 Building wheel for sqlalchemy (setup.py) ... done Created wheel for sqlalchemy: filename=SQLAlchemy-1.3.13-cp36-cp36m-linux_x86_64.whl size=1217169 sha256=849fd60745e8a83a712003f7750ef69c8d684a7f79bb2b7ad98bd088b0fd7619 Stored in directory: /root/.cache/pip/wheels/b3/35/98/4c9cb3fd63d21d5606b972dd70643769745adf60e622467b71 Building wheel for combo (setup.py) ... done Created wheel for combo: filename=combo-0.1.1-cp36-none-any.whl size=42111 sha256=693192608b2c333d5b9c651fcbf73c3a4147218ee9a81beb460b835388a5441d Stored in directory: /root/.cache/pip/wheels/55/ec/e5/a2331372c676c467e70c6646e646edf6997d5c4905b8c0f5e6 Building wheel for suod (setup.py) ... done Created wheel for suod: filename=suod-0.0.4-cp36-none-any.whl size=2167157 sha256=6b0adcda39706695e2d95182ad75eeeef30dc5e119c7a2bbe96d82aac2cd0bf8 Stored in directory: /root/.cache/pip/wheels/57/55/e5/a4fca65bba231f6d0115059b589148774b41faea25b3f2aa27 Building wheel for imagehash (setup.py) ... done Created wheel for imagehash: filename=ImageHash-4.1.0-py2.py3-none-any.whl size=291990 sha256=3cbd089c07c48233aed130c2e7b60085f7da9ae8f78695830ad7615c7ddf0d30 Stored in directory: /root/.cache/pip/wheels/07/1c/dc/6831446f09feb8cc199ec73a0f2f0703253f6ae013a22f4be9 Successfully built pyLDAvis pyod htmlmin funcy databricks-cli querystring-parser prometheus-flask-exporter sqlalchemy combo suod imagehash Installing collected packages: tangled-up-in-unicode, imagehash, visions, confuse, phik, tqdm, htmlmin, pandas-profiling, threadpoolctl, scikit-learn, yellowbrick, funcy, pyLDAvis, azure-core, cryptography, isodate, msrest, azure-storage-blob, gorilla, databricks-cli, python-editor, Mako, sqlalchemy, alembic, querystring-parser, smmap, gitdb, gitpython, prometheus-flask-exporter, websocket-client, docker, gunicorn, mlflow, lightgbm, kmodes, combo, suod, pyod, zope.interface, DateTime, catboost, datefinder, pycaret Found existing installation: tqdm 4.41.1 Uninstalling tqdm-4.41.1: Successfully uninstalled tqdm-4.41.1 Found existing installation: pandas-profiling 1.4.1 Uninstalling pandas-profiling-1.4.1: Successfully uninstalled pandas-profiling-1.4.1 Found existing installation: scikit-learn 0.22.2.post1 Uninstalling scikit-learn-0.22.2.post1: Successfully uninstalled scikit-learn-0.22.2.post1 Found existing installation: yellowbrick 0.9.1 Uninstalling yellowbrick-0.9.1: Successfully uninstalled yellowbrick-0.9.1 Found existing installation: SQLAlchemy 1.3.18 Uninstalling SQLAlchemy-1.3.18: Successfully uninstalled SQLAlchemy-1.3.18 Found existing installation: lightgbm 2.2.3 Uninstalling lightgbm-2.2.3: Successfully uninstalled lightgbm-2.2.3 Successfully installed DateTime-4.3 Mako-1.1.3 alembic-1.4.2 azure-core-1.7.0 azure-storage-blob-12.3.2 catboost-0.24 combo-0.1.1 confuse-1.3.0 cryptography-3.0 databricks-cli-0.11.0 datefinder-0.7.1 docker-4.2.2 funcy-1.14 gitdb-4.0.5 gitpython-3.1.7 gorilla-0.3.0 gunicorn-20.0.4 htmlmin-0.1.12 imagehash-4.1.0 isodate-0.6.0 kmodes-0.10.2 lightgbm-2.3.1 mlflow-1.10.0 msrest-0.6.18 pandas-profiling-2.8.0 phik-0.10.0 prometheus-flask-exporter-0.15.4 pyLDAvis-2.1.2 pycaret-2.0 pyod-0.8.1 python-editor-1.0.4 querystring-parser-1.2.4 scikit-learn-0.23.2 smmap-3.0.4 sqlalchemy-1.3.13 suod-0.0.4 tangled-up-in-unicode-0.0.6 threadpoolctl-2.1.0 tqdm-4.48.2 visions-0.4.4 websocket-client-0.57.0 yellowbrick-1.1 zope.interface-5.1.0
#Importing libraries
import pandas as pd
import numpy as np
import pycaret
Dataset information: The datasets contains transactions made by credit cards in September 2013 by european cardholders. This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.
It contains only numerical input variables which are the result of a PCA transformation.
This dataset was a part of Kaggle Competition too, where the participants needed to predict wether the transaction was a fraud one or normal.
Link to the competition: https://www.kaggle.com/mlg-ulb/creditcardfraud
df=pd.read_csv("/content/drive/My Drive/creditcard.csv")
df.head()
Time | V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | V11 | V12 | V13 | V14 | V15 | V16 | V17 | V18 | V19 | V20 | V21 | V22 | V23 | V24 | V25 | V26 | V27 | V28 | Amount | Class | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.0 | -1.359807 | -0.072781 | 2.536347 | 1.378155 | -0.338321 | 0.462388 | 0.239599 | 0.098698 | 0.363787 | 0.090794 | -0.551600 | -0.617801 | -0.991390 | -0.311169 | 1.468177 | -0.470401 | 0.207971 | 0.025791 | 0.403993 | 0.251412 | -0.018307 | 0.277838 | -0.110474 | 0.066928 | 0.128539 | -0.189115 | 0.133558 | -0.021053 | 149.62 | 0 |
1 | 0.0 | 1.191857 | 0.266151 | 0.166480 | 0.448154 | 0.060018 | -0.082361 | -0.078803 | 0.085102 | -0.255425 | -0.166974 | 1.612727 | 1.065235 | 0.489095 | -0.143772 | 0.635558 | 0.463917 | -0.114805 | -0.183361 | -0.145783 | -0.069083 | -0.225775 | -0.638672 | 0.101288 | -0.339846 | 0.167170 | 0.125895 | -0.008983 | 0.014724 | 2.69 | 0 |
2 | 1.0 | -1.358354 | -1.340163 | 1.773209 | 0.379780 | -0.503198 | 1.800499 | 0.791461 | 0.247676 | -1.514654 | 0.207643 | 0.624501 | 0.066084 | 0.717293 | -0.165946 | 2.345865 | -2.890083 | 1.109969 | -0.121359 | -2.261857 | 0.524980 | 0.247998 | 0.771679 | 0.909412 | -0.689281 | -0.327642 | -0.139097 | -0.055353 | -0.059752 | 378.66 | 0 |
3 | 1.0 | -0.966272 | -0.185226 | 1.792993 | -0.863291 | -0.010309 | 1.247203 | 0.237609 | 0.377436 | -1.387024 | -0.054952 | -0.226487 | 0.178228 | 0.507757 | -0.287924 | -0.631418 | -1.059647 | -0.684093 | 1.965775 | -1.232622 | -0.208038 | -0.108300 | 0.005274 | -0.190321 | -1.175575 | 0.647376 | -0.221929 | 0.062723 | 0.061458 | 123.50 | 0 |
4 | 2.0 | -1.158233 | 0.877737 | 1.548718 | 0.403034 | -0.407193 | 0.095921 | 0.592941 | -0.270533 | 0.817739 | 0.753074 | -0.822843 | 0.538196 | 1.345852 | -1.119670 | 0.175121 | -0.451449 | -0.237033 | -0.038195 | 0.803487 | 0.408542 | -0.009431 | 0.798278 | -0.137458 | 0.141267 | -0.206010 | 0.502292 | 0.219422 | 0.215153 | 69.99 | 0 |
df.shape
(284807, 31)
Our dataset contains 24000 rows and 24 columns
df.isnull().sum()
Time 0 V1 0 V2 0 V3 0 V4 0 V5 0 V6 0 V7 0 V8 0 V9 0 V10 0 V11 0 V12 0 V13 0 V14 0 V15 0 V16 0 V17 0 V18 0 V19 0 V20 0 V21 0 V22 0 V23 0 V24 0 V25 0 V26 0 V27 0 V28 0 Amount 0 Class 0 dtype: int64
There are no null values as observed from above table
Now, let's check for the count of positive and negative classes in our dataset
df["Class"].value_counts().plot.bar(legend=None)
<matplotlib.axes._subplots.AxesSubplot at 0x7f937f1abeb8>
This is a highly imablanced dataset.
We need to deal with the dataset in a correct way. When we will train our model than our model will achieve high accuracy but our trained model will predict a negative class in maximum number of cases. So, we also need to keep in mind the precision and recall score in such scenarios.
This problem is predominant in scenarios where anomaly detection is crucial like electricity pilferage, fraudulent transactions in banks, identification of rare diseases, etc. In this situation, the predictive model developed using conventional machine learning algorithms could be biased and inaccurate.
This happens because Machine Learning Algorithms are usually designed to improve accuracy by reducing the error. Thus, they do not take into account the class distribution / proportion or balance of classes.
This guide describes various approaches for solving such class imbalance problems using Pycaret.
from pycaret.classification import *
clf=setup(data=df,target='Class',fix_imbalance=True) #fix_imbalance will automaticaaly fix the imbalanced dataset by oversampling using the SMOTE method.
Setup Succesfully Completed!
Description | Value | |
---|---|---|
0 | session_id | 3458 |
1 | Target Type | Binary |
2 | Label Encoded | None |
3 | Original Data | (284807, 31) |
4 | Missing Values | False |
5 | Numeric Features | 30 |
6 | Categorical Features | 0 |
7 | Ordinal Features | False |
8 | High Cardinality Features | False |
9 | High Cardinality Method | None |
10 | Sampled Data | (199364, 31) |
11 | Transformed Train Set | (139554, 30) |
12 | Transformed Test Set | (59810, 30) |
13 | Numeric Imputer | mean |
14 | Categorical Imputer | constant |
15 | Normalize | False |
16 | Normalize Method | None |
17 | Transformation | False |
18 | Transformation Method | None |
19 | PCA | False |
20 | PCA Method | None |
21 | PCA Components | None |
22 | Ignore Low Variance | False |
23 | Combine Rare Levels | False |
24 | Rare Level Threshold | None |
25 | Numeric Binning | False |
26 | Remove Outliers | False |
27 | Outliers Threshold | None |
28 | Remove Multicollinearity | False |
29 | Multicollinearity Threshold | None |
30 | Clustering | False |
31 | Clustering Iteration | None |
32 | Polynomial Features | False |
33 | Polynomial Degree | None |
34 | Trignometry Features | False |
35 | Polynomial Threshold | None |
36 | Group Features | False |
37 | Feature Selection | False |
38 | Features Selection Threshold | None |
39 | Feature Interaction | False |
40 | Feature Ratio | False |
41 | Interaction Threshold | None |
42 | Fix Imbalance | True |
43 | Fix Imbalance Method | SMOTE |
#Uncomment the following code to compare the performance of all the classification models
#compare_models()
This link will provide you some overview of precision and recall. Link: https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall
We are creating the random forest classifier because it works really well with these types of dataset. You can have a quick view of the different models using 'compare_models()'
classifier=create_model('rf')
print(classifier)
Accuracy | AUC | Recall | Prec. | F1 | Kappa | MCC | |
---|---|---|---|---|---|---|---|
0 | 0.9996 | 0.9782 | 0.8333 | 0.9524 | 0.8889 | 0.8887 | 0.8907 |
1 | 0.9994 | 0.8722 | 0.7083 | 0.8947 | 0.7907 | 0.7904 | 0.7958 |
2 | 0.9994 | 0.9572 | 0.8750 | 0.8077 | 0.8400 | 0.8397 | 0.8404 |
3 | 0.9997 | 0.9386 | 0.8800 | 0.9565 | 0.9167 | 0.9165 | 0.9173 |
4 | 0.9994 | 0.9147 | 0.7500 | 0.8571 | 0.8000 | 0.7997 | 0.8015 |
5 | 0.9997 | 0.9574 | 0.9167 | 0.9167 | 0.9167 | 0.9165 | 0.9165 |
6 | 0.9995 | 0.9142 | 0.7917 | 0.9048 | 0.8444 | 0.8442 | 0.8461 |
7 | 0.9994 | 0.9141 | 0.7917 | 0.8261 | 0.8085 | 0.8082 | 0.8084 |
8 | 0.9994 | 0.9359 | 0.8333 | 0.8333 | 0.8333 | 0.8330 | 0.8330 |
9 | 0.9995 | 0.9358 | 0.7500 | 0.9474 | 0.8372 | 0.8370 | 0.8427 |
Mean | 0.9995 | 0.9318 | 0.8130 | 0.8897 | 0.8476 | 0.8474 | 0.8492 |
SD | 0.0001 | 0.0283 | 0.0630 | 0.0526 | 0.0432 | 0.0433 | 0.0425 |
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None, criterion='gini', max_depth=None, max_features='auto', max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=-1, oob_score=False, random_state=3458, verbose=0, warm_start=False)
# Plotting the classification report
plot_model(classifier,plot='class_report')
# Plotting the confusion matrix
plot_model(classifier,plot='confusion_matrix')