<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dominik Haitz on Medium]]></title>
        <description><![CDATA[Stories by Dominik Haitz on Medium]]></description>
        <link>https://medium.com/@d_haitz?source=rss-6533aa1281a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zXcTXgwzYBNfEiHivfNu3Q.jpeg</url>
            <title>Stories by Dominik Haitz on Medium</title>
            <link>https://medium.com/@d_haitz?source=rss-6533aa1281a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:46:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@d_haitz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Developer Hacks — Modern Command Line Tools and Advanced Git Commands]]></title>
            <link>https://medium.com/otto-group-data-works/developer-hacks-modern-command-line-tools-and-advanced-git-commands-e3724dab00a1?source=rss-6533aa1281a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3724dab00a1</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Dominik Haitz]]></dc:creator>
            <pubDate>Wed, 02 Oct 2024 11:52:06 GMT</pubDate>
            <atom:updated>2024-10-02T11:52:06.421Z</atom:updated>
            <content:encoded><![CDATA[<h3>Developer Hacks — Modern Command Line Tools and Advanced Git Commands</h3><p>Working with the terminal and with Git are among the basic techniques for developers. This article presents a modern development setup, state-of-the-art alternatives to classic shell programs, and advanced git commands. Using these tools will help you navigate your projects with greater ease and speed, allowing you to focus on what truly matters: building great software.</p><h3>Terminal Setup for Mac</h3><p>For Mac users, <a href="https://iterm2.com/">iTerm2</a> offers more features than the default Mac terminal app: split panes, hotkey windows, and extensive configurability. Personally, I use it with the <a href="https://github.com/catppuccin/iterm">Catppuccin Macchiato</a> color theme and especially like its feature of having a scratchpad terminal quickly available via hotkey:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*NTwFkvxdcBCCe6gZTGs2jA.gif" /></figure><p>To further enhance productivity, consider using <a href="https://rectangleapp.com/">Rectangle</a> for window management, <a href="https://alt-tab-macos.netlify.app/">Alt-Tab</a> for window switching and <a href="https://github.com/p0deje/Maccy">Maccy</a> for clipboard history.</p><h3>Shell: zsh and others</h3><p>Zsh, one of the most popular shells and the default on Mac, can be customized with <a href="https://github.com/ohmyzsh/ohmyzsh">oh-my-zsh</a>, a framework that simplifies the management of zsh configurations. It is often used with the <a href="https://github.com/romkatv/powerlevel10k">powerlevel10k</a> prompt for an aesthetically pleasing and informative terminal prompt. Additionally, you can enhance your zsh experience with plugins like <a href="https://github.com/zsh-users/zsh-autosuggestions">zsh-autosuggestions</a> or <a href="https://github.com/zsh-users/zsh-syntax-highlighting">zsh-syntax-highlighting</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*4r8cSdmS61g2PF2mlEL4Wg.png" /><figcaption>zsh with transient prompt (hide full prompt on past commands), git infos, syntax highlighting (valid command shown green) and auto suggestions</figcaption></figure><p>Newer, but less common shells are nushell, fish, and xonsh. To me, <a href="https://github.com/xonsh/xonsh">xonsh</a> looks particularly interesting as it allows mixing both Python and shell commands.</p><h3>Modern Shell Commands</h3><p>For many of the classic commands like <em>ls</em>, <em>cat</em>, <em>find</em> etc., there exist modern alternatives. Compared to their traditional counterparts, those shell commands are often faster (as they are often written in Rust), more colorful and more convenient, e.g. by respecting gitignore files.</p><h4>ls alternatives: eza and lsd</h4><p><a href="https://github.com/eza-community/eza"><em>eza</em></a> and <a href="https://github.com/lsd-rs/lsd"><em>lsd</em></a> both provide a better file listing with color highlighting, icons, and additional configurability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*2WV5AOMwP-tRPslH2O1zmg.png" /><figcaption>Output comparison for ls, lsd, eza and eza with additional options.</figcaption></figure><h4>bat: cat with wings</h4><p>A text file viewer, <a href="https://github.com/sharkdp/bat"><em>bat</em></a> uses line numbers, syntax highlighting and git changes. It uses paging for longer output (scroll text like with <em>less</em>), but defaults to plain printing when used for piping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*EueMtDdYdcDwsyNgWEWSWA.png" /><figcaption>Viewing an HTML file with <em>bat</em>, including syntax highlighting and Git integration.</figcaption></figure><h4>du to dust</h4><p>For showing file and folder sizes, <a href="https://github.com/bootandy/dust"><em>dust</em></a> provides a better sorted and visualized output than the classic <em>du</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osImUOKnDgNf2RvmtelA3g.png" /></figure><h4>fd: a better find</h4><p><a href="https://github.com/sharkdp/fd"><em>fd</em></a> is a modern alternative to <em>find</em> to search for files, much faster and with cleaner syntax: <em>fd PATTERN</em> instead of <em>find -iname ‘*PATTERN*’</em>.</p><h4>RIP, grep: ripgrep</h4><p>A tool for searching text in files. Using sensible default, <a href="https://github.com/BurntSushi/ripgrep"><em>ripgrep</em></a> considers gitignore rules and ignores hidden and binary files. Several times faster than the original <em>grep</em>.</p><h4>sed to sd</h4><p><em>sed</em> is typically used for replacing text. <em>sd</em> does this with an simpler syntax:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*2E_tJPE7ULJ5hj_DdDa6BA.png" /></figure><h4>htop alternatives: bottom, glances and btop</h4><p>For the classic <em>htop</em> tool to check system status, modern alternatives are for example <a href="https://github.com/ClementTsang/bottom"><em>bottom</em></a>, <a href="https://github.com/nicolargo/glances"><em>glances</em></a> or <a href="https://github.com/aristocratos/btop"><em>btop</em></a> (there exist even more). Although they largely use the same system metrics, they differ in their customizability and how the display and visualize data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfR0lWiO4X00_nwj0kxaig.png" /><figcaption>htop &amp; btop (top row), glances &amp; btm (bottom row)</figcaption></figure><h4>fzf: Fuzzy finder</h4><p><a href="https://github.com/junegunn/fzf">fzf</a> searches text with a so-called fuzzy search: It will also find slightly different spellings, with characters omitted. Text files, shell history or the output of other commands.The <a href="https://github.com/unixorn/fzf-zsh-plugin">fzf-zsh-plugin</a> lets you automatically search your command history with fzf when you type ^R.</p><h4>What the fuck</h4><p>The tool <a href="https://github.com/nvbn/thefuck">The Fuck</a> auto-corrects typing errors in console commands. Instead of going back and manually fixing your typo, just enter fuck and it will suggest the correct command (that you can immediately execute with enter):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*ly9k3dFUmoy9RaPSQPXgYw.png" /></figure><h3>Advanced git commands</h3><p>Apart from everyday’s commands — git init, clone, git pull, add, commit, git push , diff or show — there exist some git features which can make your life considerably easier in many situations. Let me give a git commands overview of these helpful features.</p><h4>(Global) config options</h4><p>A few helpful options are (set with <em>git config –global</em>):</p><ul><li><em>merge.conflictstyle zdiff3</em>: For merge conflicts, display the original lines in addition to the diverged versions</li><li><em>pull.rebase true</em>: Always rebase your local branch if the upstream branch has additional commits</li><li><em>push.autoSetupRemote true</em>: When running <em>git push</em> on a local branch, always set up the corresponding remote branch</li><li><em>core.pager delta</em>: Use <a href="https://github.com/dandavison/delta">delta</a> to display output of e.g. git diff</li></ul><h4>Nicer log with git oneline</h4><p>The output of <em>git log</em> can be shortened with <em>git log — pretty=oneline</em> and made even prettier with custom formatting, e.g. (s<a href="https://ma.ttias.be/pretty-git-log-in-one-line/">ource</a>):</p><p><em>git log — graph — pretty=format:’%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)&lt;%an&gt;%Creset’ — abbrev-commit</em></p><p>This can be set to an alias via <em>git config — global alias.logline “log –graph …</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*a8mpW0g7HyXUeWT-JZT17Q.png" /><figcaption>Output of <em>git log </em>with custom formatting</figcaption></figure><h4>Cleaning up a feature branch with interactive rebase</h4><p>Git’s interactive rebase can be used edit commit history by merging commits, editing their messages or dropping some of them. This is convenient when e.g. cleaning up a feature branch before merging. <em>git rebase -i</em> opens a lists of commits in an editor where you can change the default “pick” to whatever command fits. See a detailed description <a href="https://about.gitlab.com/blog/2020/11/23/keep-git-history-clean-with-interactive-rebase/">here</a>.</p><h4>Hunting bugs with git bisect</h4><p>Imagine the following scenario: Your code is not working properly, but you can’t figure out what’s wrong. You just know that a week ago, everything ran fine.</p><p>Git bisect can help you pinpoint the exact commit a bug was introduced. You run <em>git bisect start</em>, mark the good and bad commits (with <em>git bisect good / git bisect bad &lt;hash&gt;</em>). Git will automatically switch to the commit in the middle. You can mark it as good or bad and then git will switch again to commit in the middle between the good and bad commit, repeating this process until the guilty commit was found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*13-d1HqrvNKo5PXcDuNiRQ.png" /><figcaption>Starting with a bad (step 1) and good (step 2) commit, <em>git bisect </em>will jump to the commits in between (steps 3 and 4) to narrow down the commit where an error was introduced (step 5).</figcaption></figure><h4>Finding lost stuff with git reflog</h4><p>Accidentally deleted a branch or lost a commit? Don’t worry, these are still in git! You can use <em>git reflog</em> to get a list of previous commits/branches you were on and find the hash of the deleted object (and then use git checkout to restore it):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLGqRXebDARie7rza_jIjw.png" /></figure><h3>Switching to another branch with git worktree</h3><p>Let’s say you’re working on a feature, and you have many edited or new files. Suddenly, you have to fix a commit on the main branch. You’d typically either commit and stash your changes and switch to main or create another copy of the entire repository somewhere else to have a clean state.</p><p>With git worktree, you can create another worktree, something like “a copy of the repo inside your repo”, which you can delete after making changes:</p><pre>git worktree add ./fix-critical-bug main<br># go to the fix-critical-bug directory, commit and push solution on the main branch<br>git worktree remove ./fix-critical-bug<br># return to working on your feature</pre><p>In conclusion, by integrating these modern command line tools and advanced Git commands into your workflow, you can enhance your productivity and focus on crafting software that creates real business value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3724dab00a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/otto-group-data-works/developer-hacks-modern-command-line-tools-and-advanced-git-commands-e3724dab00a1">Developer Hacks — Modern Command Line Tools and Advanced Git Commands</a> was originally published in <a href="https://medium.com/otto-group-data-works">Otto Group data.works</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer Hacks — Modern Command Line Tools and Advanced Git Commands]]></title>
            <link>https://medium.com/otto-tech/developer-hacks-modern-command-line-tools-and-advanced-git-commands-2985cf5f1bb8?source=rss-6533aa1281a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2985cf5f1bb8</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[shell]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[tech-blog]]></category>
            <dc:creator><![CDATA[Dominik Haitz]]></dc:creator>
            <pubDate>Tue, 01 Oct 2024 12:17:42 GMT</pubDate>
            <atom:updated>2024-10-01T12:17:42.477Z</atom:updated>
            <content:encoded><![CDATA[<h3>Developer Hacks — Modern Command Line Tools and Advanced Git Commands</h3><p>Working with the terminal and with Git are among the basic techniques for developers. This article presents a modern development setup, state-of-the-art alternatives to classic shell programs, and advanced git commands. Using these tools will help you navigate your projects with greater ease and speed, allowing you to focus on what truly matters: building great software.</p><h3>Terminal Setup for Mac</h3><p>For Mac users, <a href="https://iterm2.com/">iTerm2</a> offers more features than the default Mac terminal app: split panes, hotkey windows, and extensive configurability. Personally, I use it with the <a href="https://github.com/catppuccin/iterm">Catppuccin Macchiato</a> color theme and especially like its feature of having a scratchpad terminal quickly available via hotkey:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*NTwFkvxdcBCCe6gZTGs2jA.gif" /></figure><p>To further enhance productivity, consider using <a href="https://rectangleapp.com/">Rectangle</a> for window management, <a href="https://alt-tab-macos.netlify.app/">Alt-Tab</a> for window switching and <a href="https://github.com/p0deje/Maccy">Maccy</a> for clipboard history.</p><h3>Shell: zsh and others</h3><p>Zsh, one of the most popular shells and the default on Mac, can be customized with <a href="https://github.com/ohmyzsh/ohmyzsh">oh-my-zsh</a>, a framework that simplifies the management of zsh configurations. It is often used with the <a href="https://github.com/romkatv/powerlevel10k">powerlevel10k</a> prompt for an aesthetically pleasing and informative terminal prompt. Additionally, you can enhance your zsh experience with plugins like <a href="https://github.com/zsh-users/zsh-autosuggestions">zsh-autosuggestions</a> or <a href="https://github.com/zsh-users/zsh-syntax-highlighting">zsh-syntax-highlighting</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*4r8cSdmS61g2PF2mlEL4Wg.png" /><figcaption>zsh with transient prompt (hide full prompt on past commands), git infos, syntax highlighting (valid command shown green) and auto suggestions</figcaption></figure><p>Newer, but less common shells are nushell, fish, and xonsh. To me, <a href="https://github.com/xonsh/xonsh">xonsh</a> looks particularly interesting as it allows mixing both Python and shell commands.</p><h3>Modern Shell Commands</h3><p>For many of the classic commands like <em>ls</em>, <em>cat</em>, <em>find</em> etc., there exist modern alternatives. Compared to their traditional counterparts, those shell commands are often faster (as they are often written in Rust), more colorful and more convenient, e.g. by respecting gitignore files.</p><h4>ls alternatives: eza and lsd</h4><p><a href="https://github.com/eza-community/eza"><em>eza</em></a> and <a href="https://github.com/lsd-rs/lsd"><em>lsd</em></a> both provide a better file listing with color highlighting, icons, and additional configurability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*2WV5AOMwP-tRPslH2O1zmg.png" /><figcaption>Output comparison for ls, lsd, eza and eza with additional options.</figcaption></figure><h4>bat: cat with wings</h4><p>A text file viewer, <a href="https://github.com/sharkdp/bat"><em>bat</em></a> uses line numbers, syntax highlighting and git changes. It uses paging for longer output (scroll text like with <em>less</em>), but defaults to plain printing when used for piping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*EueMtDdYdcDwsyNgWEWSWA.png" /><figcaption>Viewing an HTML file with <em>bat</em>, including syntax highlighting and Git integration.</figcaption></figure><h4>du to dust</h4><p>For showing file and folder sizes, <a href="https://github.com/bootandy/dust"><em>dust</em></a> provides a better sorted and visualized output than the classic <em>du</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osImUOKnDgNf2RvmtelA3g.png" /></figure><h4>fd: a better find</h4><p><a href="https://github.com/sharkdp/fd"><em>fd</em></a> is a modern alternative to <em>find</em> to search for files, much faster and with cleaner syntax: <em>fd PATTERN</em> instead of <em>find -iname ‘*PATTERN*’</em>.</p><h4>RIP, grep: ripgrep</h4><p>A tool for searching text in files. Using sensible default, <a href="https://github.com/BurntSushi/ripgrep"><em>ripgrep</em></a> considers gitignore rules and ignores hidden and binary files. Several times faster than the original <em>grep</em>.</p><h4>sed to sd</h4><p><em>sed</em> is typically used for replacing text. <em>sd</em> does this with an simpler syntax:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*2E_tJPE7ULJ5hj_DdDa6BA.png" /></figure><h4>htop alternatives: bottom, glances and btop</h4><p>For the classic <em>htop</em> tool to check system status, modern alternatives are for example <a href="https://github.com/ClementTsang/bottom"><em>bottom</em></a>, <a href="https://github.com/nicolargo/glances"><em>glances</em></a> or <a href="https://github.com/aristocratos/btop"><em>btop</em></a> (there exist even more). Although they largely use the same system metrics, they differ in their customizability and how the display and visualize data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfR0lWiO4X00_nwj0kxaig.png" /><figcaption>htop &amp; btop (top), glances &amp; btm (bottom)</figcaption></figure><h4>fzf: Fuzzy finder</h4><p><a href="https://github.com/junegunn/fzf">fzf</a> searches text with a so-called fuzzy search: It will also find slightly different spellings, with characters omitted. Text files, shell history or the output of other commands.The <a href="https://github.com/unixorn/fzf-zsh-plugin">fzf-zsh-plugin</a> lets you automatically search your command history with fzf when you type ^R.</p><h4>What the fuck</h4><p>The tool <a href="https://github.com/nvbn/thefuck">The Fuck</a> auto-corrects typing errors in console commands. Instead of going back and manually fixing your typo, just enter fuck and it will suggest the correct command (that you can immediately execute with enter):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*ly9k3dFUmoy9RaPSQPXgYw.png" /></figure><h3>Advanced git commands</h3><p>Apart from everyday’s commands — git init, clone, git pull, add, commit, git push, diff or show — there exist some git features which can make your life considerably easier in many situations. Let me give a git commands overview of these helpful features.</p><h4>(Global) config options</h4><p>A few helpful options are (set with <em>git config –global</em>):</p><ul><li><em>merge.conflictstyle zdiff3</em>: For merge conflicts, display the original lines in addition to the diverged versions</li><li><em>pull.rebase true</em>: Always rebase your local branch if the upstream branch has additional commits</li><li><em>push.autoSetupRemote true</em>: When running <em>git push</em> on a local branch, always set up the corresponding remote branch</li><li><em>core.pager delta</em>: Use <a href="https://github.com/dandavison/delta">delta</a> to display output of e.g. git diff</li></ul><h4>Nicer log with git oneline</h4><p>The output of <em>git log</em> can be shortened with <em>git log — pretty=oneline</em> and made even prettier with custom formatting, e.g. (s<a href="https://ma.ttias.be/pretty-git-log-in-one-line/">ource</a>):</p><p><em>git log — graph — pretty=format:’%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)&lt;%an&gt;%Creset’ — abbrev-commit</em></p><p>This can be set to an alias via <em>git config — global alias.logline “log –graph …</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*a8mpW0g7HyXUeWT-JZT17Q.png" /><figcaption>Output of <em>git log </em>with custom formatting</figcaption></figure><h4>Cleaning up a feature branch with interactive rebase</h4><p>Git’s interactive rebase can be used edit commit history by merging commits, editing their messages or dropping some of them. This is convenient when e.g. cleaning up a feature branch before merging. <em>git rebase -i</em> opens a lists of commits in an editor where you can change the default “pick” to whatever command fits. See a detailed description <a href="https://about.gitlab.com/blog/2020/11/23/keep-git-history-clean-with-interactive-rebase/">here</a>.</p><h4>Hunting bugs with git bisect</h4><p>Imagine the following scenario: Your code is not working properly, but you can’t figure out what’s wrong. You just know that a week ago, everything ran fine.</p><p>Git bisect can help you pinpoint the exact commit a bug was introduced. You run <em>git bisect start</em>, mark the good and bad commits (with <em>git bisect good / git bisect bad &lt;hash&gt;</em>). Git will automatically switch to the commit in the middle. You can mark it as good or bad and then git will switch again to commit in the middle between the good and bad commit, repeating this process until the guilty commit was found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*13-d1HqrvNKo5PXcDuNiRQ.png" /><figcaption>Starting with a bad (step 1) and good (step 2) commit, <em>git bisect </em>will jump to the commits in between (steps 3 and 4) to narrow down the commit where an error was introduced (step 5).</figcaption></figure><h4>Finding lost stuff with git reflog</h4><p>Accidentally deleted a branch or lost a commit? Don’t worry, these are still in git! You can use <em>git reflog</em> to get a list of previous commits/branches you were on and find the hash of the deleted object (and then use git checkout to):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLGqRXebDARie7rza_jIjw.png" /></figure><h4>Switching to another branch with git worktree</h4><p>Let’s say you’re working on a feature, and you have many edited or new files. Suddenly, you have to fix a commit on the main branch. You’d typically either commit and stash your changes and switch to main or create another copy of the entire repository somewhere else to have a clean state.</p><p>With git worktree, you can create another worktree, something like “a copy of the repo inside your repo”, which you can delete after making changes:</p><pre>git worktree add ./fix-critical-bug main<br># go to the fix-critical-bug directory, commit and push solution on the main branch<br>git worktree remove ./fix-critical-bug<br># return to working on your feature</pre><p>In conclusion, by integrating these modern command line tools and advanced Git commands into your workflow, you can enhance your productivity and focus on crafting software that creates real business value.</p><p>👉 Read this one and more interesting articles on our <a href="https://www.otto.de/jobs/en/technology/techblog/blogpost/developer-hacks-modern-command-line-tools-and-advanced-git-commands.php">Tech Blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2985cf5f1bb8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/otto-tech/developer-hacks-modern-command-line-tools-and-advanced-git-commands-2985cf5f1bb8">Developer Hacks — Modern Command Line Tools and Advanced Git Commands</a> was originally published in <a href="https://medium.com/otto-tech">OTTO Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cyberpunk Style with Matplotlib]]></title>
            <link>https://medium.com/data-science/cyberpunk-style-with-matplotlib-f47404c9d4c5?source=rss-6533aa1281a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f47404c9d4c5</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[cyberpunk]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dominik Haitz]]></dc:creator>
            <pubDate>Sat, 28 Mar 2020 16:05:41 GMT</pubDate>
            <atom:updated>2020-03-29T21:05:41.691Z</atom:updated>
            <content:encoded><![CDATA[<p>Futuristic neon glow for your next data visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WfxjwsuEf9tzJu07.png" /></figure><p><em>Update 2020–03–29: There’s now a Python package to conveniently apply this style, see </em><a href="https://github.com/dhaitz/mplcyberpunk"><em>here</em></a><em>. Install via</em>pip install mplcyberpunk</p><h3>1 — The Basis</h3><p>Let’s make up some numbers, put them in a Pandas dataframe and plot them:</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt</pre><pre>df = pd.DataFrame({&#39;A&#39;: [1, 3, 9, 5, 2, 1, 1],<br>                   &#39;B&#39;: [4, 5, 5, 7, 9, 8, 6]})</pre><pre>df.plot(marker=&#39;o&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wRegampkiDj_Q_VE.png" /></figure><h3>2 — The Darkness</h3><p>Not bad, but somewhat ordinary. Let’s customize it by using Seaborn’s dark style, as well as changing background and font colors:</p><pre>plt.style.use(&quot;seaborn-dark&quot;)</pre><pre>for param in [&#39;figure.facecolor&#39;, &#39;axes.facecolor&#39;, &#39;savefig.facecolor&#39;]:<br>    plt.rcParams[param] = &#39;#212946&#39;  # bluish dark grey</pre><pre>for param in [&#39;text.color&#39;, &#39;axes.labelcolor&#39;, &#39;xtick.color&#39;, &#39;ytick.color&#39;]:<br>    plt.rcParams[param] = &#39;0.9&#39;  # very light grey</pre><pre>ax.grid(color=&#39;#2A3459&#39;)  # bluish dark grey, but slightly lighter than background</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kbBP7WEi9BHEjEgf.png" /></figure><h3>3 — The Light</h3><p>It looks more interesting now, but we need our colors to shine more against the dark background:</p><pre>fig, ax = plt.subplots()<br>colors = [<br>    &#39;#08F7FE&#39;,  # teal/cyan<br>    &#39;#FE53BB&#39;,  # pink<br>    &#39;#F5D300&#39;,  # yellow<br>    &#39;#00ff41&#39;, # matrix green<br>]<br>df.plot(marker=&#39;o&#39;, ax=ax, color=colors)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1JA72L3pGzGpkfvI.png" /></figure><h3>4 — The Glow</h3><p>Now, how to get that neon look? To make it shine, we <em>redraw the lines multiple times</em>, with low alpha value and slighty increasing linewidth. The overlap creates the glow effect.</p><pre>n_lines = 10<br>diff_linewidth = 1.05<br>alpha_value = 0.03</pre><pre>for n in range(1, n_lines+1):</pre><pre>    df.plot(marker=&#39;o&#39;,<br>            linewidth=2+(diff_linewidth*n),<br>            alpha=alpha_value,<br>            legend=False,<br>            ax=ax,<br>            color=colors)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E7WySsBePeA89OVi.png" /></figure><h3>5 — The Finish</h3><p>For some more fine tuning, we color the area below the line (via ax.fill_between) and adjust the axis limits.</p><p>Here’s the full code:</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br></pre><pre>plt.style.use(&quot;dark_background&quot;)</pre><pre>for param in [&#39;text.color&#39;, &#39;axes.labelcolor&#39;, &#39;xtick.color&#39;, &#39;ytick.color&#39;]:<br>    plt.rcParams[param] = &#39;0.9&#39;  # very light grey</pre><pre>for param in [&#39;figure.facecolor&#39;, &#39;axes.facecolor&#39;, &#39;savefig.facecolor&#39;]:<br>    plt.rcParams[param] = &#39;#212946&#39;  # bluish dark grey</pre><pre>colors = [<br>    &#39;#08F7FE&#39;,  # teal/cyan<br>    &#39;#FE53BB&#39;,  # pink<br>    &#39;#F5D300&#39;,  # yellow<br>    &#39;#00ff41&#39;,  # matrix green<br>]<br></pre><pre>df = pd.DataFrame({&#39;A&#39;: [1, 3, 9, 5, 2, 1, 1],<br>                   &#39;B&#39;: [4, 5, 5, 7, 9, 8, 6]})</pre><pre>fig, ax = plt.subplots()</pre><pre>df.plot(marker=&#39;o&#39;, color=colors, ax=ax)</pre><pre># Redraw the data with low alpha and slighty increased linewidth:<br>n_shades = 10<br>diff_linewidth = 1.05<br>alpha_value = 0.3 / n_shades</pre><pre>for n in range(1, n_shades+1):</pre><pre>    df.plot(marker=&#39;o&#39;,<br>            linewidth=2+(diff_linewidth*n),<br>            alpha=alpha_value,<br>            legend=False,<br>            ax=ax,<br>            color=colors)</pre><pre># Color the areas below the lines:<br>for column, color in zip(df, colors):<br>    ax.fill_between(x=df.index,<br>                    y1=df[column].values,<br>                    y2=[0] * len(df),<br>                    color=color,<br>                    alpha=0.1)</pre><pre>ax.grid(color=&#39;#2A3459&#39;)</pre><pre>ax.set_xlim([ax.get_xlim()[0] - 0.2, ax.get_xlim()[1] + 0.2])  # to not have the markers cut off<br>ax.set_ylim(0)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WfxjwsuEf9tzJu07.png" /></figure><p>If this helps you or if you have constructive criticism, I’d be happy to hear about it! Please contact me via <a href="https://dhaitz.github.io/">here</a> or <a href="https://twitter.com/d_haitz">here</a>. Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f47404c9d4c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/cyberpunk-style-with-matplotlib-f47404c9d4c5">Cyberpunk Style with Matplotlib</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Regularization for Machine Learning Models]]></title>
            <link>https://medium.com/data-science/regularization-for-machine-learning-models-9173c2e90449?source=rss-6533aa1281a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9173c2e90449</guid>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[regularization]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dominik Haitz]]></dc:creator>
            <pubDate>Thu, 25 Jul 2019 15:21:18 GMT</pubDate>
            <atom:updated>2019-07-25T15:21:18.269Z</atom:updated>
            <content:encoded><![CDATA[<h4>L1 Lasso, L2 Ridge and L1+L2 Elastic Net regularization explained</h4><p>A common problem in machine learning is <strong>overfitting</strong>, where a model falsely generalizes noise in the training data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*rrjJaesIzo46XTT3y-SYXA.gif" /><figcaption><a href="https://towardsdatascience.com/over-fitting-and-regularization-64d16100f45c">https://towardsdatascience.com/over-fitting-and-regularization-64d16100f45c</a></figcaption></figure><p>A popular approach to remedy this problem and make the model more robust is <strong>regularization</strong>: A <strong>penalty term</strong> is added to the algorithm’s loss function. This changes the model’s weights which result from minimizing the loss function.</p><p>The most popular regularization techniques are<strong> Lasso, Ridge</strong> (aka Tikhonov) and <strong>Elastic Net</strong>. For the exemplary case of simple linear regression with only one <strong>weight parameter <em>w</em></strong> (the slope of the linear fit), their penalty terms look like this (including a <strong>scaling parameter <em>λ</em></strong>):</p><ul><li><strong>Lasso (L1)</strong>: <em>λ·|w|</em></li><li><strong>Ridge (L2)</strong>: <em>λ·w</em>²</li><li><strong>Elastic Net (L1+L2)</strong>: <em>λ</em>₁<em>·|w| + λ</em>₂<em>·w</em>²</li></ul><p>The different terms have different effects: <strong>Compared to L1, the quadratic L2 regularization becomes negligible at small weights (close to zero), but stronger at large weights.</strong> This leads to the following behaviours, casually phrased:</p><ul><li><strong>Linear Regression without regularization</strong>: “I go along with everything.”</li><li><strong>Lasso</strong>: “I’m skeptic at first, but go along with significant trends.”</li><li><strong>Ridge</strong>: “I’m easily convinced, but somewhat sluggish.”</li><li><strong>Elastic Net:</strong> “I’m feeling somewhere between Ridge and LASSO.”</li></ul><p>Let’s see what this looks like <strong>in practice</strong>. For ten random numbers, we’ll perform a linear fit with each of the four methods above (using an increased lambda parameter for Ridge, for demonstration purposes):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zd_AED0qTGrne104M7Ng6Q.png" /></figure><p>Of course, you’ll get a minor random trend in the data, which is unquestioningly picked up by Linear Regression. Ridge regression also picks up the trend, yet weaker. For Lasso and Elastic Net, the linear L1 penalty term is high enough to force the weight (i.e. the slope) to zero when minimizing the loss function.</p><p>Now, we add a small linear component to the data points and re-run the fitting procedures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdft4k41rndb8C3sHWfAmw.png" /></figure><p>This is already enough for Lasso to not fully “ignore” the slope coefficient anymore.</p><p>If we further increase the added linear component, we get this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5q_D2lnfkw47ikO4khu2nA.png" /></figure><p>The Lasso and Elastic Net now almost fully “accept” the significant trend, while for Ridge, the quadratic penalty term leads to a lower slope.</p><p>Do you miss anything? Share your opinion or questions in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9173c2e90449" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/regularization-for-machine-learning-models-9173c2e90449">Regularization for Machine Learning Models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating and Deploying a Python Machine Learning Service]]></title>
            <link>https://medium.com/data-science/creating-and-deploying-a-python-machine-learning-service-a06c341f020f?source=rss-6533aa1281a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a06c341f020f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Dominik Haitz]]></dc:creator>
            <pubDate>Mon, 08 Apr 2019 13:37:25 GMT</pubDate>
            <atom:updated>2019-04-08T13:37:25.435Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build a hate speech detection system with scikit-learn and deploy it via Docker on Heroku.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6x9WsXQZmDB0S4odp-KkQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jontyson?utm_source=medium&amp;utm_medium=referral">Jon Tyson</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Imagine you’re the moderator of a message board or comment section. You don’t want to read everything your users write online, yet you want to be alerted in case a discussion turns sour or people start spewing racial slurs all over the place. So, you decide to build yourself an automated system for hate speech detection.</p><p>Text classification via machine learning is an obvious choice of technology. However, turning model prototypes into working services has proven to be a widespread challenge. To help bridge this gap, this four-step tutorial illustrates an exemplary deployment workflow for a hate speech detection app:</p><ol><li>Train and persist a prediction model with <strong>scikit-learn</strong></li><li>Create an API endpoint with <strong>firefly</strong></li><li>Create a <strong>Docker</strong> container for this service</li><li>Deploy the container on <strong>Heroku</strong></li></ol><p>The code for this project is available <a href="https://github.com/dhaitz/python-sklearn-firefly-docker-heroku">here</a>.</p><h3>1. Create prediction model</h3><h4>Dataset</h4><p>The approach is based on the paper <a href="https://arxiv.org/abs/1703.04009"><em>Automated Hate Speech Detection and the Problem of Offensive Language</em></a> by Davidson, Warmsley, Macy and Weber. Their results are based on more than 20 000 labelled tweets, which are available on the <a href="https://github.com/t-davidson/hate-speech-and-offensive-language">corresponding Github page</a>.</p><p>The .csv file is loaded as a dataframe:</p><pre>import pandas as pd<br>import re</pre><pre><strong>df</strong> = pd.read_csv(&#39;labeled_data.csv&#39;, usecols=[&#39;class&#39;, &#39;tweet&#39;])</pre><pre><strong>df</strong>[&#39;tweet&#39;] = <strong>df</strong>[&#39;tweet&#39;].apply(lambda tweet: re.sub(&#39;[^A-Za-z]+&#39;, &#39; &#39;, tweet.lower()))</pre><p>The last line cleans the tweet column by converting all text to lowercase and removing non-alphabetic characters.</p><p>Result:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72b0f1d1c019279e96cd13df1725f6b7/href">https://medium.com/media/72b0f1d1c019279e96cd13df1725f6b7/href</a></iframe><p>The class attribute can assume three category values: 0 for hate speech, 1 for offensive language and 2 for neither.</p><h4>Model training</h4><p>We have to convert our predictors, i.e. the tweet text, into a numeric representation before we can train a machine learning classifier. We can use scikit-learn’s <strong>TfidfVectorizer</strong> for this task, which transforms texts into a matrix of term-frequency times inverse document-frequency (tf-idf) values, suitable for machine learning. Additionally, we can remove <strong>stop words</strong> (common words such as <em>the</em>, <em>is</em>, …) from the processing.</p><p>For text classification, support vector machines (<strong>SVMs</strong>) are a reliable choice. As they are binary classifiers, we will use a <strong>One-Vs-Rest</strong> strategy, where for each category an SVM is trained to separate this category from all others.</p><p>Both text vectorization and SVM training can be performed in one command by using scikit-learn’s <strong>Pipeline</strong> feature and defining the respective steps:</p><pre>from sklearn.pipeline import make_pipeline<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.multiclass import OneVsRestClassifier<br>from sklearn.svm import SVC<br>from stop_words import get_stop_words</pre><pre><strong>clf</strong> = make_pipeline(<br>    TfidfVectorizer(stop_words=get_stop_words(&#39;en&#39;)),<br>    OneVsRestClassifier(SVC(kernel=&#39;linear&#39;, probability=True))<br>)</pre><pre><strong>clf</strong> = <strong>clf</strong>.fit(X=<strong>df</strong>[&#39;tweet&#39;], y=<strong>df</strong>[&#39;class&#39;])</pre><p>Now, the performance of the model should be evaluated, e.g. using a cross-validation approach to calculate classification metrics. However, as this tutorial focusses on model deployment, we will skip this step (<strong>never do this in an actual project</strong>). The same goes for parameter tuning or additional techniques of natural language processing which are described in the <a href="https://arxiv.org/abs/1703.04009">original paper</a>.</p><h4>Test the model</h4><p>We can now try a test text and have the model predict the probabilities:</p><pre><strong>text</strong> = &quot;I hate you, please die!&quot;<strong><br>clf</strong>.predict_proba([<strong>text</strong>.lower()])</pre><pre># Output:<br>array([0.64, 0.14, 0.22])</pre><p>The numbers in the array correspond to the probabilities for the three categories (hate speech, offensive language, neither).</p><h4>Model persistence</h4><p>Using the joblib module, we can save the model as a binary object to disk. This will allow us to load and use the model in an application.</p><pre>from sklearn import externals</pre><pre><strong>model_filename</strong> = &#39;hatespeech.joblib.z&#39;<br>externals.joblib.dump(<strong>clf</strong>, <strong>model_filename</strong>)</pre><h3>2. Create REST API</h3><h4>Create API endpoint</h4><p>The python file app.py loads the model and defines a simple module-level function which wraps the call to the model’s predict_proba function:</p><pre>from sklearn import externals</pre><pre><strong>model_filename</strong> = &#39;hatespeech.joblib.z&#39;<br><strong>clf </strong>= externals.joblib.load(model_filename)</pre><pre>def predict(<strong>text</strong>):<br>    probas = <strong>clf</strong>.predict_proba([<strong>text</strong>.lower()])[0]<br>    return {&#39;hate speech&#39;: probas[0],<br>           &#39;offensive language&#39;: probas[1],<br>           &#39;neither&#39;: probas[2]}</pre><p>Now, we use <a href="https://github.com/rorodata/firefly">firefly</a>, a lightweight python module for <em>function as a service</em>. For advanced configuration or use in a production environment, Flask or Falcon might be a better choice as they’re well established with a large community. For rapid prototyping, we’re fine with firefly.</p><p>We’ll use firefly on the command line to bind the predict function to port 5000 on localhost:</p><pre><strong>$</strong> firefly app.predict --bind 127.0.0.1:5000</pre><h4>Test API locally</h4><p>Via curl, we can make a POST request to the created endpoint and obtain a prediction:</p><pre><strong>$</strong> curl -d &#39;{&quot;text&quot;: &quot;Please respect each other.&quot;}&#39; \ http://127.0.0.1:5000/predict</pre><pre># Output:<br>{&quot;hate speech&quot;: 0.04, &quot;offensive language&quot;: 0.31, &quot;neither&quot;: 0.65}</pre><p>Of course, in a full-fledged real application there would be much more additional features (logging, input and output validation, exception handling, …) and work steps (documentation, versioning, testing, monitoring, …), but here we’re merely deploying a simple prototype.</p><h3>3. Create a Docker container</h3><p>Why Docker? A Docker container runs an application in an isolated environment, with all dependencies included, and can be shipped as an image, thus simplifying service setup and scaling.</p><h4>Build image</h4><p>We have to configure the contents and start-actions of our container in a file named Dockerfile:</p><pre>FROM python:3.6<br>RUN pip install scikit-learn==0.20.2  firefly-python==0.1.15<br>COPY app.py hatespeech.joblib.z ./</pre><pre>CMD firefly app.predict --bind 0.0.0.0:5000<br>EXPOSE 5000</pre><p>The first three lines are about taking python:3.6 as base image, additionally installing scikit-learn and firefly (the same versions as in the development environment) and copying the app and model files inside. The latter two lines tell Docker the command which is executed when a container is started and that port 5000 should be exposed.</p><p>The build process that creates the image hatespeechdetect is started via:</p><pre><strong>$</strong> docker build . -t hatespeechdetect</pre><h4>Run Container</h4><p>The run command starts a container, derived from an image. Additionally, we’re binding the containers’s port 5000 to the host’s port 3000 via the -p option:</p><pre><strong>$</strong> docker run -p 3000:5000 -d hatespeechdetect</pre><h4>Use prediction service</h4><p>Now, we can send a request and obtain a prediction:</p><pre><strong>$</strong> curl -d &#39;{&quot;text&quot;: &quot;You are fake news media! Crooked!&quot;}&#39; \ http://127.0.0.1:3000/predict</pre><pre># Output:<br>{&quot;hate speech&quot;: 0.08, &quot;offensive language&quot;: 0.76, &quot;neither&quot;: 0.16}</pre><p>In this example, the container runs locally. Of course the actual purpose is to keep it running at a permanent location, and possibly scale the service by starting multiple containers in an enterprise cluster.</p><h3>4. Deploy as an Heroku app</h3><p>A way to make the app publicly available to others is using a platform as a service such as <strong>Heroku</strong>, which supports Docker and offers a free basic membership. To use it, we have to register an account and install the <a href="https://devcenter.heroku.com/articles/heroku-cli,">Heroku CLI</a>.</p><p>Heroku’s application containers expose a dynamic port, which requires an edit in our Dockerfile: We have to change port 5000 to the environment variable PORT:</p><pre>CMD firefly app.predict --bind 0.0.0.0:$PORT</pre><p>After this change, we are ready for deployment. On the command line, we log in to heroku (which will prompt us for credentials in the browser) and create an app named hate-speech-detector:</p><pre><strong>$</strong> heroku login</pre><pre><strong>$ </strong>heroku create hate-speech-detector</pre><p>Then we log in to the container registry. heroku container:push will build an image based on the Dockerfile in the current directory and send it to the Heroku Container registry. After that, we can release the image to the app:</p><pre><strong>$</strong> heroku container:login</pre><pre><strong>$</strong> heroku container:push web --app hate-speech-detector</pre><pre><strong>$</strong> heroku container:release web --app hate-speech-detector</pre><p>As before, the API can be addressed via curl. However, this time, the service is not running locally, but is available to the world!</p><pre><strong>$</strong> curl -d ‘{“text”: “You dumb idiot!”}’ https://hate-speech-detector.herokuapp.com/predict</pre><pre># Output:<br>{&quot;hate speech&quot;: 0.26, &quot;offensive language&quot;: 0.68, &quot;neither&quot;: 0.06}</pre><p>Now, scaling the app would be just a few clicks or commands away. Also, the service needs to be connected to the message board, the trigger threshold needs to be set and an alerting implemented.</p><p>Hopefully this tutorial will help you deploy your own machine learning models and apps. Got any additional ideas? Share your opinion in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a06c341f020f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/creating-and-deploying-a-python-machine-learning-service-a06c341f020f">Creating and Deploying a Python Machine Learning Service</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jupyter Notebook Best Practices]]></title>
            <link>https://medium.com/data-science/jupyter-notebook-best-practices-f430a6ba8c69?source=rss-6533aa1281a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f430a6ba8c69</guid>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dominik Haitz]]></dc:creator>
            <pubDate>Wed, 27 Mar 2019 14:04:09 GMT</pubDate>
            <atom:updated>2019-06-25T12:56:16.557Z</atom:updated>
            <content:encoded><![CDATA[<h4>Concise advice to use Jupyter notebooks more effectively.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CXz0JwDBjuoQVIeO" /><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Caveat: The advice in this article refers to the original Jupyter notebook. While much of the advice can be adapted to JupyterLab, the popular notebook extensions can’t.</p><h4>Table of Contents</h4><p><a href="https://medium.com/p/f430a6ba8c69#87fc">1. Structure your Notebook</a></p><p><a href="https://medium.com/p/f430a6ba8c69#f5af">2. Refactor &amp; outsource code into modules</a></p><p><a href="https://medium.com/p/f430a6ba8c69#e75a">3. Be curious about productivity hacks</a></p><p><a href="https://medium.com/p/f430a6ba8c69#dfd1">4. Embrace reproducibility</a></p><p><a href="https://medium.com/p/f430a6ba8c69#cafb">5. Further reading</a></p><p><a href="https://medium.com/p/f430a6ba8c69#2d44">Conclusion</a></p><h4>1. Structure your Notebook</h4><ul><li>Give your notebook a <strong>title</strong> (H1 header) and a meaningful <strong>preamble</strong> to describe its purpose and contents.</li><li>Use <strong>headings and documentation</strong> in Markdown cells to structure your notebook and explain your workflow steps. Remember: You’re not only doing this for your colleagues or your successor, but also for your future self.</li><li>The <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/toc2/README.html">toc2</a> extension can automatically create heading numbers and a <strong>Table of Contents</strong>, both in a sidebar (optionally a floating window) and in a markdown cell. The highlighting indicates your current position in the document — this will help you keep oriented in long notebooks.</li><li>The <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/collapsible_headings/readme.html"><strong>Collapsible Headings</strong></a> extension allows you to hide entire sections of code, thereby letting you focus on your current workflow stage.</li><li><a href="https://towardsdatascience.com/set-your-jupyter-notebook-up-right-with-this-extension-24921838a332">This <strong>default template</strong> extension</a> causes notebooks to not be created empty, but with a default structure and common imports. Also, it will repeatedly ask you to change the name from Untitled.ipynb to something meaningful.</li><li>The Jupyter <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/snippets/README.html">snippets</a> extension allows you to conveniently <strong>insert often needed code blocks</strong>, e.g. your typical import statements.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*QTp1c_ODrL7GiqsiIOeOzA.png" /><figcaption>Using a Jupyter notebook template (which sets up default imports and structure) and the Table of Contents (toc2) extension, which automatically numbers headings. The Collapsible Headings extension enables hiding of section contents by clicking the grey triangles next to the headings.</figcaption></figure><h4>2. Refactor &amp; outsource code into modules</h4><ul><li>After you’ve written plain code in cells to get ahead quickly, acquire the habit of <strong>turning stable code into</strong> <strong>functions</strong> and <strong>move them to a dedicated module</strong>. This makes your notebook more readable and is incredibly helpful when productionizing your workflow. This:</li></ul><pre>df = pd.read_csv(filename)<br>df.drop( ...<br>df.query( ...<br>df.groupby( ...</pre><p>becomes this:</p><pre>def load_and_preprocess_data(filename):<br>   &quot;&quot;&quot;DOCSTRING&quot;&quot;&quot;<br>   # do stuff<br>   # ...<br>   return df</pre><p>and finally this:</p><pre>import dataprep<br>df = dataprep.load_and_preprocess_data(filename)</pre><ul><li>If you edit a module file, Jupyter’s <a href="https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html">autoreload</a> extension <strong>reloads imported modules</strong>:</li></ul><pre>%load_ext autoreload<br>%autoreload</pre><ul><li>Use <a href="https://pypi.org/project/ipytest/">ipytest</a> for <strong>testing</strong> inside notebooks.</li><li>Use a proper <strong>IDE</strong>, e.g. <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>. Learn about its features for efficient debugging, refactoring and testing.</li><li><strong>Stick to the standards of good coding</strong> — think <strong><em>Clean Code</em> principles </strong>and <a href="https://www.python.org/dev/peps/pep-0008/"><strong>PEP8</strong></a>. Use meaningful variable and function names, comment sensibly, modularize your code and don’t be too lazy to refactor.</li></ul><h4>3. Be curious about productivity hacks</h4><ul><li>Learn the <a href="https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook"><strong>Jupyter Keyboard Shortcuts</strong></a>. Print the list and hang it on the wall next to your screen.</li><li>Get to know <strong>Jupyter extensions:</strong> <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/codefolding/readme.html">Codefolding</a>, <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/hide_input_all/readme.html">Hide input all</a>, <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/varInspector/README.html">Variable Inspector</a>, <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/splitcell/readme.html">Split Cells Notebook,</a> zenmode and <a href="https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions.html">many more</a>.</li><li><a href="https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20Basics.html">Jupyter Widgets</a> (sliders, buttons, dropdown-menus, …) allow you to build <strong>interactive GUIs</strong>.</li><li>The <a href="https://github.com/tqdm/tqdm#ipython-jupyter-integration">tqdm</a> library provides a convenient<strong> progress bar</strong>.</li></ul><h4>4. Embrace reproducibility</h4><ul><li><strong>Version Control:</strong> Learn to use git — there are many great tutorials out there.</li><li>Depending on your project and purpose, it might be reasonable to use a <a href="https://github.com/kynan/nbstripout"><strong>git pre-commit hook that removes notebook output</strong></a>. It will make commits and diffs more readable, but might discard output (plots etc.) you actually want to store.</li><li>Run your notebook in a <strong>dedicated (conda) environment</strong>. Store the requirements.txt file in the git repository alongside your notebooks and modules. This will help you reproduce your workflow as well as facilitate transitioning into a production environment.</li></ul><h4>5. Further reading</h4><ul><li><a href="https://florianwilhelm.info/2018/11/working_efficiently_with_jupyter_lab/">Working efficiently with JupyterLab Notebooks</a></li><li><a href="https://towardsdatascience.com/bringing-the-best-out-of-jupyter-notebooks-for-data-science-f0871519ca29">Bringing the best out of Jupyter Notebooks for Data Science</a></li><li><a href="https://towardsdatascience.com/jupyter-notebook-hints-1f26b08429ad">Boosting Your Jupyter Notebook Productivity</a></li><li>… and of course Joel Grus’ famous <a href="https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/">I Don’t Like Notebooks</a></li></ul><h3>Conclusion</h3><p>Good software engineering practices, structuring and documenting your workflow as well as customizing Jupyter to your personal taste will increase your notebook productivity and sustainability.</p><p>I’m happy to hear your own tips and your feedback in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f430a6ba8c69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/jupyter-notebook-best-practices-f430a6ba8c69">Jupyter Notebook Best Practices</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>