All posts by Marijn van Vliet

Integrating Vim and jupyter-qtconsole

Generally, for those looking for a comfortable Python data analysis environment, I recommend using the Spyder IDE. It aims to mimic MATLAB’s excellent GUI and does a pretty good job of it.

However, I personally shy away from IDE’s and use Vim as my text editor for basically everything. This tip is for those few that are like me that need a comfortable data analysis environment centered around Vim. This can be achieved by coupling Vim to Jupyter’s QtConsole.

The idea is to have an interactive console open for quick commands (“plot this data”, “what is the mean of this thing?”, etc.) and a Vim instance where you work on analysis scripts and so forth.

When you start QtConsole, two processes are created: a Jupyter kernel running IPython and a Jupyter client that connects to the kernel and builds a GUI to interact with it. The jupyter-vim Vim plugin adds the ability to connect Vim to the Jupyter kernel started by QtConsole. The situation is thus that there is one Jupyter kernel with two connected clients. Both clients have full access to the state of the kernel, can send commands to it, receive results back and can monitor traffic between other clients and the kernel. This setup allows us to make Vim and the QtConsole cooperate nicely.

[embedyt] https://www.youtube.com/watch?v=h59cbg4HqpY[/embedyt]

Installation

Of course, you need to have a python installation with the jupyter-qtconsole package installed. You need version 4.4 or newer. You also need the jupyter-vim plugin. You can find it at: https://github.com/jupyter-vim/jupyter-vim.

For the jupyter-vim plugin to work properly, you need Vim version 8 or above with Python support compiled in (which it usually is these days). Furthermore, whatever Python installation Vim is linked against (usually the system Python) needs to have the jupyter package installed. To find our which Python Vim is using, open Vim and use the command:

:pythonx import sys; print(sys.path)

In my case, Vim tells me /usr/lib/python3.6/site-packages is the first item in its Python path, so my Vim is using the system python located at /usr/bin/python. If this is also the case for you, use your system’s package manager to install Jupyter. If this is not the case for you, or if there is no such official package, use PIP to install it. However, make sure to the PIP binary that is part of the Python installation that Vim is using! (In my case that would be /usr/bin/pip).

Configuration

To make QtConsole output whatever text you send it, you can set the following in your ~/.jupyter/jupyter_qtconsole_config.py:

c.ConsoleWidget.include_other_output = True

This instructs QtConsole to echo the communication between the IPython kernel and the Vim. This causes the commands send by Vim to the kernel and the response of the kernel to be shown in the QtConsole window.

Usage

If everything is setup correctly, you should be able to open a QtConsole and a Vim instance running side by side. Then, in Vim, open a Python file and give the following command:

:JupyterConnect

Vim will attempt to connect to the IPython kernel. If this is successful you can use (among others) the following keybindings to send Python code to be executed on the kernel:

<localleader>R    Run the file
<localleader>I    Import the file
<localleader>d    Change the working directory of the Jupyter kernel
                  to the directory containing this file
<localleader>X    Execute this code cell (delimit with ##, #%% or # )
<localleader>E    Execute this line
<localleader>e    Use with any text-object to execute it. Or in visual 
                  mode: execute selected code

You will see the communication between Vim and the kernel displayed in the QtConsole, prefixed with [remote].

Becoming friends with Microsoft Word

Love it, hate it, the fact is that everybody has to deal with Microsoft Word at some point. And the reason for that is that everybody needs to write documents at some point and Word is the program of choice to do it in for many people out there.

Why is there so much hate about Word? I think it is because managing the layout of a document is actually quite tricky. It is an art. If you are frustrated with Word, it is likely you are underestimating the task you are wishing to perform. Continue reading Becoming friends with Microsoft Word

Add a splash of color to your command line environment

By default, much of the text in a Linux terminal is just black letters on a white background. While it has a certain peaceful aesthetic about it, this wall of text is needlessly difficult to “parse” with your eyes. Everyone I know that remained sane while using the terminal for extended periods of time has configured the thing to their personal taste.

As you probably know already, your terminal can display colors. Color can be leveraged to add another dimension of information to the text on your screen and provide helpful marks so you can easily see what’s going on.

Continue reading Add a splash of color to your command line environment

Version control features in Word: remain sane when collaborating on a document

You are probably aware of the track-changes feature of Microsoft Word. When you send a document to someone else to review, they can use the feature to track any changes they made to the document. After you get the document back, you can then go over the changes one by one and decide whether to accept them or not.

But what if you *gasp* keep working on the document after you’ve sent it? By the time you get the comments back from the reviewer, they will be comments on an outdated version of your document! Alternatively, you may have sent the document to two or more people to comment on. Now you have multiple different versions that all contain useful changes.

And so the grand manual labor of redoing all of the changes in the new document begins… or so you may think.

Luckily, Word has some version control tricks, so there is a better way! Continue reading Version control features in Word: remain sane when collaborating on a document

The most useful key on your keyboard: TAB

It has occurred to me that not everyone has been using a command line since the DOS era. Some may have only encountered the command line very recently as they make their first steps into data analysis.

This one is for you new people. This is the most important thing you should now if you want to survive on the command line. You don’t need to type as much as you think you do! Introducing the most worn out key on my keyboard: TAB.
Continue reading The most useful key on your keyboard: TAB

doit: a Python alternative to make

In an earlier post, I demonstrated the wonderfulness that is having a build-system and introduced the venerable make utility. You can use it to add plumbing to your analysis pipeline that keeps track of which analysis steps have been run for which subjects and whether some scripts have changed and should be run again. When properly implemented, it can save a lot of head aces, especially when deadlines are approaching, by making sure that everything is always ‘up do date’.

However, make is not the easiest tool to wrap your head around. The syntax is archaic, we needed to add “phony” targets and use a custom made, magical-looking function to iterate over subjects. So let me introduce you to another tool, called doit, which can be easier to use, especially if you are familiar with Python syntax. Continue reading doit: a Python alternative to make

make: intelligent plumbing for your analysis pipeline

Most of our data analysis happens in scripts. When these scripts grow and become more awesome, they also tend to take longer to run. We’re talking hours, sometimes even days, of computation time here. Naturally, it becomes unpractical to re-run the entire thing every time we change something. Instead, we only re-run the parts of it that changed. This post is about a tool that has been around for ages and can incredibly helpful here, but you may not have heard of it: make! Continue reading make: intelligent plumbing for your analysis pipeline