How to write a custom appender in log4j2?

/* package declaration, imports... */

@Plugin(name = "CustomListAppender",
        category = Core.CATEGORY_NAME,
        elementType = Appender.ELEMENT_TYPE,
        printObject = true)
public final class CustomListAppender extends AbstractAppender {

    // for storing the log events
    private List<LogEvent> events = new ArrayList<>();

    protected CustomListAppender(
            String name,
            Filter filter,
            Layout<? extends Serializable> layout,
            boolean ignoreExceptions) {
        super(name, filter, layout, ignoreExceptions);
    }

    @Override
    public void append(LogEvent event) {
        if (event instanceof MutableLogEvent) {
            events.add(((MutableLogEvent) event).createMemento());
        } else {
            events.add(event);
        }
    }

    public List<LogEvent> getEvents() {
        return events;
    }

    @PluginFactory
    public static CustomListAppender createAppender(
            @PluginAttribute("name") String name,
            @PluginElement("Layout") Layout<? extends Serializable> layout,
            @PluginElement("Filter") Filter filter) {
        if (name == null) {
            LOGGER.error("No name provided for TestLoggerAppender");
            return null;
        }

        if (layout == null) layout = PatternLayout.createDefaultLayout();

        return new CustomListAppender(name, filter, layout, true);
    }
}

Our CustomListAppender extends AbstractAppender, because that implements a lot of the methods from the Appender interface for us that we would otherwise have to implement ourselves.

The @Plugin annotation identifies this class a plugin that should be picked up by the PluginManager:

  • The name attribute defines the name of the appender that can be used in the configuration.
  • The category attribute should be "Core", because “Core plugins are those that are directly represented by an element in a configuration file, such as an Appender, Layout, Logger or Filter” (source). And we are creating an appender.
  • The elementType attribute defines which type of element in the Core category this plugin should be. In our case, "appender".
  • The printObject attribute defines whether our custom plugin class defines a useful toString() method. We do, because the AbstractAppender class we’re extending is taking care of that for us.

We implement the Appender#append(LogEvent) method to add each event to our events list. If the LogEvent happens to be mutable, we must take care to create an immutable copy of the event, otherwise subsequent log events will overwrite it (we will get a list of, say, three log events that are all referencing the same object). We also add a simple getter method to retrieve all log events.

For the PluginManager to create our custom plugin, it needs a way to instantiate it. log4j2 uses a factory method for that, indicated by the annotation @PluginFactory. An appender contains attributes, such as a name, and other elements, such as layouts and filters. To allow for these, we use the corresponding annotations @PluginAttribute to indicate that a parameter represents an attribute, and @PluginElement to indicate that a parameter represents an element.

To log errors that might occur during this setup, we can make use of the StatusLogger. This logger is available as LOGGER, and is defined in one of the parents of our custom plugin, AbstractLifeCycle. (The level of log messages that should be visible can be adjusted in the <Configuration status="warn" ...> element.)

Configuration

Configuration:
  packages: com.relentlesscoding.logging.plugins
  status: warn
  appenders:
    Console:
      name: STDOUT
    CustomListAppender:
      name: MyVeryOwnListAppender

  Loggers:
    logger:
      -
        name: com.relentlesscoding.logging
        level: info
        AppenderRef:
          ref: MyVeryOwnListAppender
    Root:
      level: error
      AppenderRef:
        ref: STDOUT

The packages attribute on the Configuration element indicates the package that should be scanned by the PluginManager for custom plugins during initialization.

How to use our custom list appender?

private CustomListAppender appender;

@Before
public void setupLogging() {
    LoggerContext context = LoggerContext.getContext(false);
    Configuration configuration = context.getConfiguration();
    appender = configuration.getAppender("MyVeryOwnListAppender");
    appender.getEvents().clear();
}

When we run tests now, we are able to see all logged events by calling appender.getEvents(). Before each test, we take care to clear the list of the previous log statements.

Unit test log4j2 log output

Sometimes you want to test if certain log output gets generated when certain events happen in your application. Here is how I unit test that using log4j2 (version 2.11.0).

Use LoggerContextRule to get to your ListAppender quickly

If you are using JUnit 4, then the quickest solution would be one that is used by log4j2 itself:

import org.apache.logging.log4j.junit.LoggerContextRule;
/* other imports */

public class LogEventTest {
    private static ListAppender appender;

    @ClassRule
    public static LoggerContextRule init = new LoggerContextRule("log4j2-test.yaml");

    @BeforeClass
    public static void setupLogging() {
        appender = init.getListAppender("List");
    }

    @Before
    public void clearAppender() {
        appender.clear();
    }

    @Test
    public void someMethodShouldLogAnError() {
        // setup test and invoke logic
        List<LogEvent> logEvents = appender.getEvents();
        List<String> errors = logEvents.stream()
                .filter(event -> event.getLevel().equals(Level.ERROR))
                .map(event -> event.getMessage().getFormattedMessage())
                .collect(Collectors.toList());

        // we logged at least one event of level error
        assertThat(errors.size(), is(greaterThanOrEqualTo(1)));

        // log event message should contain "wrong" for example
        assertThat(errors, everyItem(containsString("wrong")));
    }
}

The LoggerContextRule provides methods that come in handy while testing. Here we use the getListAppender(...) method to get access to an appender that uses a list to store all log events. Before each test, we clear the list, so we have a clean slate for new log events. The test invokes the code-under-test, requests the log events from the appender and filters them so that we only have the error log events left. Then we that at least one error log message was captured and that it contains the word “wrong”.

Use the LoggerContext

Instead of using the class rule (which lets you conveniently pass it the file name of the configuration), you could also use the LoggerContext:

@BeforeClass
public static void setupLogging() {
    LoggerContext context = LoggerContext.getContext(false);
    Logger logger = context.getLogger("com.relentlesscoding");
    appender = (ListAppender) logger.getAppenders().get("List");
}

This might be your only option if you are working with JUnit 5 (and eventually you will want to migrate to that). In JUnit 5 we can’t use LoggerContextRule anymore, because @Rules don’t longer exist (they were replaced with an extension mechanism that works differently and log4j2 doesn’t provide such an extension currently).

Create a working configuration

To get the examples working, we need to define an appender called “List” and a logger in our log4j2 configuration.

log4j2-test.yaml

Configuration:
  status: warn
  name: TestConfig
  appenders:
    Console:
      name: STDOUT
    List:
      name: List

  Loggers:
    logger:
      -
        name: com.relentlesscoding
        AppenderRef:
          ref: List
    Root:
      level: info AppenderRef:
        ref: STDOUT

This configuration will send log events occurring in the package com.relentlesscoding and sub-packages to the appender with the name “List” (which is of type ListAppender). This configuration is defined in YAML, but you can use XML, JSON or the properties format as well.

Maven dependencies

To get the LoggerContextRule in JUnit 4 working, you need the following:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.11.0</version>
    <type>test-jar</type>
</dependency>

To get the YAML log4j2 configuration working, you need:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.9.5</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.dataformat</groupId>
    <artifactId>jackson-dataformat-yaml</artifactId>
    <version>2.9.5</version>
</dependency>

JMockit fakes

If you are working with microservices, you may find that you are having a lot of dependencies on other services. If you want your tests to be autonomous, that is, running in isolation of those dependencies, you can either use a tool like WireMock, or you can use a mocking framework.

Recently, I came across JMockit, a mocking framework for Java. To mock external dependencies, you can use so-called “fakes”.

Say you use a service that knows the credentials of the customers of your e-store, and you make requests to this service by using a driver provided by this service, say CredentialsHttpService. In your development environment (and Jenkins), you don’t have access to a running service. A solution to this would be a fake implementation of CredentialsHttpService, where we would mock the methods that we actually call from our code.

public class CredentialsHttpService {
    ...
    public Optional<CustomerAccount> getCustomerAccount(String id) {
        // does HTTP request to service
    }
    ...
}

In our test code, we can now implement the fake:

public class ServiceTest {
    @Tested
    Service service;

    @Test
    void test() {
        new MockUp<CredentialsHttpService>() {
            final static AtomicInteger counter = new AtomicInteger();

            @Mock
            Optional<CustomerAccount> getCustomerAccount(String id) {
                return Optional.of(CustomerAccount.builder()
                                        .accountNumber(counter.incrementAndGet())
                                        .build());
            }
        }

        service.doFoo();  // eventually invokes our fake
                          // getCustomerAccount() implementation
    }
}

The class to be faked is the type parameter of the MockUp class. The fake CredentialsHttpService will only mock the methods that are annotated with @Mock. All other methods of the faked class will have their real implementation. Mocked methods are not restricted by access modifiers: the method may have a private, protected, package-private or public access modifier, as long as the method name and number of parameters and parameter types are the same.

There is a lot of other things that fakes can do in JMockit, see the documentation.

Bash’ magic space

What does the “magic space” do?

Given the following:

$ find -wholename '*/path/to/file' -print -quit
$ man rm
$ rm -fv !-2:2

In the last line, feedback would be appreciated to see if we are indeed going to delete the second argument of two commands back. If you set Bash’ so-called “magic space”, history expansion will take place right away after typing a space after !-2:2:

$ rm -fv '*/path/to/file'

How to enable the magic space?

Put the following in your ~/.inputrc:

$if Bash
    Space: magic-space
$endif

Start a new session, or use bind -f ~/.inputrc to put the changes in effect immediately.

Other ways to achieve the same

You could also enable shopt -s histverify, which will perform the history expansion and give you another opportunity to modify the command before executing it. This requires you to press enter, though.

Unit test Grails GORM’s formulas

The code for this post is part of my PomoTimer project and can be found on GitHub.

The domain class

We have a domain class Project that models a project that one is working on when doing a particular work session. It records the name of the project (“Writing a blog post”), a status (“active”, “completed”), a creation time, the total time spent on this project and the user the project belongs to.

The total time is a derived field: it calculates the time spent on the work sessions that belong to this project. It should only calculate the work sessions that have status “done”. Derived fields can be defined in Grails by so-called formulas.

package com.relentlesscoding.pomotimer

class Project {
    String name
    ProjectStatus status
    Date creationtime
    Integer totaltime
    User user

    static constraints = {
        name blank: false, nullable: false, maxSize: 100, unique: true
        status blank: true, nullable: false, display: true
        creationtime blank: false, nullable: false
        totaltime blank: false, nullable: false, min: 0
        user blank: false, nullable: false, display: true
    }

    static mapping = {
        totaltime formula: '(select ifnull(sum(select d.seconds from Duration d where ws.duration_id = d.id), 0) from Work_Session ws where ws.project_id = id and ws.status_id in (select st.id from Work_Session_Status st where st.name = \'done\'))'
    }

    String toString() { return name }
}

Testing the formula

So how do we test a formula? Initially, I tried to write integration tests (grails create-integration-test <className> that create a class with the grails.testing.mixin.Integration and grails.transaction.Rollback annotations), but these suffer from the limitation that each feature method starts a new transaction, and formula’s aren’t updated until after the transaction is committed (which is never because the transaction is always rolled back). This effectively makes it impossible for an integration test to check whether a formula does what it is supposed to do.

The solution is to write a test specification that extends HibernateSpec, which allows us to fully use Hibernate in our unit tests.

package com.relentlesscoding.pomotimer

import grails.test.hibernate.HibernateSpec

class ProjectTotaltimeSpec extends HibernateSpec {

    def 'adding a work session should increase total time'() {
        given: 'a new project'
        def projectStatus = new ProjectStatus(name: 'foo', description: 'bar')
        def user = new User(firstName: 'foo',
                            lastName: 'bar',
                            userName: 'baz',
                            email: 'q@w.com',
                            password: 'b' * 10)
        def project = new Project(name: 'a new project',
                                  status: projectStatus,
                                  creationtime: new Date(),
                                  totaltime: 0,
                                  user: user)

        expect: 'total time of the new project is 0 seconds'
        project.totaltime == 0

        when: 'adding a completed work session to the new project'
        def duration = new Duration(seconds: 987)
        def done = new WorkSessionStatus(name: 'done', description: 'done')
        new WorkSession(starttime: new Date(),
                        duration: duration,
                        project: project,
                        status: done
                .save(failOnError: true, flush: true)
        // NOTE: a refresh is required to update the domain object
        project.refresh()

        then: 'total time is equal to duration of completed work session'
        project.totaltime == 987
    }
}

Import contacts (vCards) into Nextcloud

TL;DR

Export your contacts from Google in vCard version 3 format, split the contacts file and use cadaver to upload all files individually to your address book.

The struggle

Last week, I did a fresh install of Lingeage OS 14.1 on my OnePlus X and decided not to install any GApps. I have been slowly moving away from using Google services and, having found replacements in the form of open-source apps or web interfaces, I felt confident I would be able to use my phone without a Google Play Store or Play Services. (F-Droid is now my sole source of apps.)

To tackle the problem of storing contacts and a calendar that could be synced, I installed a Nextcloud instance on a Raspberry Pi 3. Having installed DAVdroid, I got my phone to sync contacts with Nextcloud, but not all of them: it would stop synchronizing after some 120 contacts, while I had more than 400.

I decided to try a different approach, so I exported the contacts on my phone in vCard format and tried to upload them to Nextcloud using the aptly named application "contacts" for this. However, this also failed unexpectedly. I’m using Nextcloud version 12.0.3 and version 2.0.1 of the contact app, but it refuses to accept vCard version 2.1 (HTTP response code 415: Unsupported media type). This, naturally, is the version Android 6 uses to export contacts.

After some searching, I found out that if you go to contacts.google.com, you can download your contacts in vCards version 3. Problem fixed? Well, not so fast: importing 400+ contacts into Nextcloud using the web interface on a Raspberry Pi 3 with an SD card for storage will take a long time. In fact, it never finished over the course of a couple of hours (!), so I needed yet another approach.

Fortunately, you can approach your Nextcloud instance through the WebDAV protocol using tools such as cadaver:

$ cadaver https://192.168.1.14/nextcloud/remote.php/dav

Storing your credentials in a .netrc file in your home directory will enable cadaver to verify your identity without prompting, making it suitable for scripting:

machine 192.168.1.14
login foo
password correcthorsebatterystaple

cadaver allows you to traverse the directories of the remote file system over WebDAV. To put a single local contacts file (from your working machine) to the remote Raspberry Pi, you could tell it to:

dav:/nextcloud/remote.php/dav/> cd addressbooks/users/{username}/{addressbookname}
dav:/nextcloud/remote.php/dav/addressbooks/users/foo/Contacts/> put /home/foo/all.vcf all.vcf

I had a single vcf file with 400+ contacts in them, but after uploading it this way, only a single contact was being displayed. Apparently, the Nextcloud’s contacts app assumes a single vcf file contains only a single contact. New challenge: we need to split this single vcf file containing multiple contacts into separate files that we can then upload to Nextcloud.

To split the contacts, we can use awk:

BEGIN {
    RS="END:VCARD\r?\n"
    FS="\n"
}
{
    command = "echo -n $(pwgen 20 1).vcf"
    command | getline filename
    close(command)
    print $0 "END:VCARD" > filename
}

This separates the contacts on the record separator END:VCARD and generates a random filename to store the individual contact in. (I also wrote a Java program to do the same thing, which is faster when splitting large files).

Obviously, it would be convenient now if we could upload all these files in one go. cadaver does provides the mput action to do so, but I did not get it to work with wildcards. So instead, I created a file with put commands:

for file in *.vcf; do
    echo "put $(pwd)/$file addressbooks/users/foo/Contacts/$file" >> commands
done

And then provided this as input to cadaver:

$ cadaver http://192.168.1.14/nextcloud/remote.php/dav <<< $(cat commands)

This may take a while (it took around an hour for 400+ contacts), but at least you get to see the progress as each request is made and processed. And voilà, all the contacts are displayed correctly in Nextcloud.

Always-on VPN and captive portals

I was attending a meetup that was taking place in a bar in Utrecht. The first thing you want to do is to make a connection to the internet and get started. The location used a captive portal, however. You know: you have the name of the wireless network (SSID) and the password, but when you try to open any web page, you are directed towards a login page where you have to accept the terms and conditions of whoever is operating the network.

But what if you use an always-on VPN? You cannot connect to the network, because your MAC and IP address are not whitelisted yet by the operator. And you cannot get to the login page, because you do not allow any traffic outside your VPN.

The captive portal page.
The captive portal page.

UFW

I use ufw (uncomplicated firewall) as my firewall of choice, mainly because the alternative, iptables, always looked too complicated and ufw served its purpose. The rules I have for ufw are currently:

# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), deny (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
Anywhere on wlp1s0         ALLOW IN    192.168.178.0/24

Anywhere                   ALLOW OUT   Anywhere on tun0-unrooted
192.168.178.0/24           ALLOW OUT   Anywhere on wlp1s0
1194                       ALLOW OUT   Anywhere on wlp1s0
Anywhere (v6)              ALLOW OUT   Anywhere (v6) on tun0-unrooted
1194 (v6)                  ALLOW OUT   Anywhere (v6) on wlp1s0

By default, I deny all incoming and outgoing connections. I only allow incoming connections from hosts on the same network. As for outgoing connections, I only allow them to other hosts over the LAN (to any port), and everywhere else only over port 1194 (the OpenVPN port). (wlp1s0 is the name of my wireless interface, tun0-unrooted of the VPN tunnel. These rules were inspired by this Arch Linux article).

So we need to allow some traffic outside of the VPN tunnel to accept the terms and conditions and to register our machine at the captive portal. The best thing to do would be to allow a single, trusted application to access this portal, one that would be used exclusively for this task. If you would allow you regular browser to bypass the VPN, it would send all kind of traffic over the untrusted network for the rest of the world to freely sniff around in (think add-ons, other browser tabs, automatic updates). So we would need a dedicated web browser for this task. I’m on Linux using Firefox as my default browser, so GNOME Web would be a good choice for this purpose. (Gnome Web was previously known as Epiphany, and is still available under that name on a lot of distributions) .

First, we need to determine what kind of traffic we want to allow. The application will need to have outbound access to ports 80 (HTTP) and 443 (HTTPS) for web traffic, and it will also need to be able to resolve domain names using DNS, so port 53 should also be opened.

UFW Profiles

However, it’s not that easy to allow one particular application access to the internet if you use UFW. When you look at the man page for UFW, you see you can specify “apps”. Apps (or application profiles) are basically just text files in INI-format that live in the /etc/ufw/applications.d/ folder.

To list all (predefined) profiles:

# ufw app list

To create a profile for our purposes, we put the following in a file called ufw-webbrowser:

[Epiphany]
title=Epiphany
description=Epiphany web browser
ports=80/tcp|443/tcp|53

The “ports” field is clarified in the man page:

The ‘ports’ field may specify a ‘|’-separated list of ports/protocols where the protocol is optional. A comma-separated list or a range (specified with ‘start:end’) may also be used to specify multiple ports, in which case the protocol is required.

In our case we allow TCP traffic over ports 80 (HTTP) and 443 (HTTPS) and both UDP and TCP traffic over 53 (DNS). We can now use this profile:

# ufw insert 1 allow out to any app Epiphany
Rule added
Rule added (v6)
# ufw status verbose
[...snip...]
To                         Action      From
--                         ------      ----
Anywhere on wlp1s0         ALLOW IN    192.168.178.0/24

80/tcp (Epiphany)          ALLOW OUT   Anywhere
443/tcp (Epiphany)         ALLOW OUT   Anywhere
53 (Epiphany)              ALLOW OUT   Anywhere
Anywhere                   ALLOW OUT   Anywhere on tun0-unrooted
192.168.178.0/24           ALLOW OUT   Anywhere on wlp1s0
1194                       ALLOW OUT   Anywhere on wlp1s0
80/tcp (Epiphany (v6))     ALLOW OUT   Anywhere (v6)
443/tcp (Epiphany (v6))    ALLOW OUT   Anywhere (v6)
53 (Epiphany (v6))         ALLOW OUT   Anywhere (v6)
Anywhere (v6)              ALLOW OUT   Anywhere (v6) on tun0-unrooted
1194 (v6)                  ALLOW OUT   Anywhere (v6) on wlp1s0
# ufw reload  # don't forget to reload firewall after making changes!
Firewall reloaded

(Note that we use insert 1 to make sure the rule is placed in the first position. With UFW, the first rule matched wins.)

Now we can use our dedicated browser to go to the captive portal page and accept the terms and conditions.

Checking traffic with wireshark

You need to be careful, however, not to use any other applications during this time. If you launch Firefox, for example, it can also use the opened ports to communicate with the outside world. I’d like to use Wireshark to see what communications are taking place during this time.

Use wireshark to see what packets are send in the open.
Use wireshark to see what packets are send in the open.

When you are registered with the WiFi provider and are done with the captive portal, you should first disable the profile again with UFW. We can do this by specifying the rule we added earlier, but prepending delete:

# ufw delete allow out to any app Epiphany

Another way to delete rules from UFW is by first doing ufw status numbered and then ufw delete <number>. However, since we have added 6 rules, this may take a while. Also, if you can’t remember the exact rule that was used, you can use ufw show added to show all added rules and their syntax.

Better solutions: beyond UFW

Now, we see that using UFW isn’t exactly ideal to deal with always-on VPN and captive portals. What if you have an email application (or something else) running in the background when you have allowed all those ports to bypass the VPN tunnel? And also, you have to enable and disable the application profile every time you encounter a captive portal. It would be better if we could allow only a single, named, demarcated application to bypass the VPN.

One solution I’ve read about makes use of what I like to call “the Android way”: every installed application is a user with its own home directory. This means that applications don’t have access to each other files, but more importantly, this gives the opportunity to allow only a specific user to access the internet outside of the VPN. This way, we could create a user epiphany that runs Gnome Web to access the captive portal.

AFWall+, an open-source Android application, uses this method to implement a pretty effective firewall. It also uses iptables as a back-end. I might have to finally bite the bullet and learn iptables after all…

Tutorial: rapid GUI development with Qt Designer and PyQt

Confession: I am the opposite of the lazy coder. I like doing things the hard way. Whether it’s developing Java in Vim without code completion, running JUnit tests on the command line (don’t forget to specify all 42 dependencies in the colon-separated classpath!), creating a LaTeX graph that looks “just right”, or writing sqlplus scripts instead of using SQL Developer (GUIs are for amateurs), I always assumed that doing so would make me a better programmer.

So when I was just a fledgling programmer, and I had to design and create some dialog windows for a side project of mine (an awesome add-on to AnkiSRS written in Python and Qt), I did what I always do: find a good resource to learn PyQt and then code everything by hand. Now, since Python is a dynamic language and I used to develop this add-on in Vim, I had no code completion whatsoever, and only the Qt documentation and that tutorial to go by. Making things even more interesting, is that the documentation on Qt is in C++, and is full of stuff like this:

Qt documentation with very readable C++ code for the Python novice.
Qt documentation with very readable C++ code for the Python novice.

Obviously, I had no idea what all the asterisks and ampersands meant and had to use good-old trial and error to see what would stick in Python. Now, on the plus side, I experimented quite a lot with the code, but not having code completion makes it really hard to learn the API. (And even now, using PyCharm, code completion will not always work, because of Python being a dynamically-typed language and all. I am still looking at the Qt documentation quite a bit.)

One thing I noticed, though, is that the guy who develops Anki, Damien Elmes, had all these .ui files lying around, and a bunch more files that read: “WARNING! All changes made to this file will be lost!”. Ugh, generated code! None of the dedicated, soul-cleansing and honest hard work that will shape you as a software developer. It goes without saying I stayed far away from that kind of laziness.

It took me some time to come around on this. One day, I actually got a job as a professional software developer and I had much less time to work on my side-projects. So, when I wanted to implement another feature for my Anki add-on, I found I had too little time to do it the old-fashioned way, and decided to give Qt Designer a go. Surprisingly, I found out it can actually help you tremendously to learn how Qt works (and also save you a bunch of time). Qt Designer allows you to visually create windows using a drag-and-drop interface, then spews out an XML representation of that GUI, which can be converted to code. That generated code will show you possibilities that would have taken a long time and a lot of StackOverflowing to figure out on your own.

So, to help other people discover the wonders of this program, here is a little tutorial on how to do RAD and how to get a dialog window up and running with Qt Designer 4 and Python 3.

Installation

First, we need to install Qt Designer. On Arch Linux, it’s part of the qt4 package and you’ll also need python-pyqt4 for the Python bindings. (On Ubuntu, you have to install both qt4-designer and the python-qt4 packages.)

Our goal

Our goal is to create a simple dialog window that has a text input field where we can type our name, and have a label that will display “Hello there, $userName”. Yes, things will be that exciting. Along the way, we will learn how to assign emitted signals to slots and how to handle events.

Creating a dialog

Fire up Qt Designer, and you will be presented with a “New form” dialog (if you do not see it, go to File > New…).

Qt Designer’s “new form” dialog.
Qt Designer’s “new form” dialog.

For this tutorial, we are going to choose a fairly small “Dialog with Buttons Bottom”:

The main window of Qt Designer.
The main window of Qt Designer.

To the left are the widgets that we can add to our freshly created dialog, to the right, from top to bottom, we see the currently added widgets, the properties of those widgets and the signals and slots currently assigned to the dialog window.

I’m going to add a text label and a so-called line editor to our widget. To make sure they will align nicely, I will put them together in a horizontal container, a QHBoxLayout:

A horizontal box layout with a text label and a line editor.
A horizontal box layout with a text label and a line editor.

In the object inspector, we can see the hierarchy of added widgets:

The object-inspector view
The object-inspector view

This is all simple drag-and-drop: we select a widget from the left pane, drag it to the desired location in the dialog and release the mouse.

Finally, we add two more label: one at the top, instructing the user what to do, and a second one near the bottom, where we soon will display our message.

Two more text labels.
Two more text labels.

Qt Designer provides an easy way to connect signals to slots. If you go to Edit > Edit Signals/Slots (or press F4) you will be presented with a graphical overview of the currently assigned signals and slots. When we start out, the button box at the bottom already emits two signals: rejected and accepted, from the Cancel and Ok button respectively:

The signals and slots of the button box.
The signals and slots of the button box.

The signals rejected() and accepted() are emitted when the cancel and OK buttons are clicked respectively, and the ground symbols indicate the object that is interested in these signals: in this case the dialog window itself. Signals are handled by slots, and so the QDialog will need to have slots for these signals. The slots (or handlers) are named reject() and accept() in this instance, and since they are default slots provided by Qt, they already exist in QDialog, so we won’t have to do anything (except if we want to override their default behavior).

What we want to do now is “catch” the textEdited signal from the line editor widget and create a slot in the dialog window that will handle it. This new slot we’ll call say_hello. So we click the line editor widget and drag a line to anywhere on the dialog window: a ground symbol should be visible:

Assigning a signal to a slot
Assigning a signal to a slot

In the window that appears now, we can select the signal that we are interested in (textEdited) and assign it to a predefined slot, or we can click on Edit… and create our own slot. Let’s do that:

Creating a new slot.
Creating a new slot.

We click the green plus sign, and type in the name of our new slot (say_hello):

Selecting the newly created slot.
Selecting the newly created slot.

The result will now look like:

All signals and slots.
All signals and slots.

In Qt Designer, you can preview your creation by going to Form > Preview… (or pressing Ctrl + R). Notice, however, that typing text in the line editor won’t do anything yet. This is because we haven’t written any implementation code for it. In the next section we will see how to do that.

You can also see the code that will be generated by going to Form > View code…, although this code is going to be in C++.

Okay, enough with designing our dialog window, let’s get to actual Python coding.

Generating code (or not)

When we save our project in Qt Designer, it will create a .ui file, which is just XML containing all the properties (widgets, sizes, signals & slots, etc.) that make up the GUI.

PyQt comes with a program, pyuic4, that can convert these .ui files to Python code. Two interesting command-line options are --preview (or -p for short), that will allow you to preview the dynamically created GUI, and --execute (or -x for short), that will generate Python code that can be executed as a stand-alone. The --output switch (or -o) allows you to specify a filename where the code will be saved.

In our case, creating a stand-alone executable is not going to work, because we have added a custom slot (say_hello) that needs to be implemented first. So we will use pyuic4 to generate the form class:

$ pyuic4 relentless_dialog.ui -o relentless_dialog.py

This is the result:

# -*- coding: utf-8 -*-

# Form implementation generated from reading ui file 'relentless_dialog.ui'
#
# Created by: PyQt4 UI code generator 4.12.1
#
# WARNING! All changes made in this file will be lost!

from PyQt4 import QtCore, QtGui

try:
    _fromUtf8 = QtCore.QString.fromUtf8
except AttributeError:
    def _fromUtf8(s):
        return s

try:
    _encoding = QtGui.QApplication.UnicodeUTF8
    def _translate(context, text, disambig):
        return QtGui.QApplication.translate(
                context, text, disambig, _encoding)
except AttributeError:
    def _translate(context, text, disambig):
        return QtGui.QApplication.translate(context, text, disambig)

class Ui_Dialog(object):
    def setupUi(self, Dialog):
        Dialog.setObjectName(_fromUtf8("Dialog"))
        Dialog.resize(250, 320)
        self.buttonBox = QtGui.QDialogButtonBox(Dialog)
        self.buttonBox.setGeometry(QtCore.QRect(10, 270, 221, 41))
        self.buttonBox.setOrientation(QtCore.Qt.Horizontal)
        self.buttonBox.setStandardButtons(
                QtGui.QDialogButtonBox.Cancel|QtGui.QDialogButtonBox.Ok)
        self.buttonBox.setObjectName(_fromUtf8("buttonBox"))
        self.horizontalLayoutWidget = QtGui.QWidget(Dialog)
        self.horizontalLayoutWidget.setGeometry(
                QtCore.QRect(10, 40, 231, 80))
        self.horizontalLayoutWidget.setObjectName(
                _fromUtf8("horizontalLayoutWidget"))
        self.horizontalLayout = QtGui.QHBoxLayout(
                self.horizontalLayoutWidget)
        self.horizontalLayout.setMargin(0)
        self.horizontalLayout.setObjectName(_fromUtf8("horizontalLayout"))
        self.label = QtGui.QLabel(self.horizontalLayoutWidget)
        self.label.setObjectName(_fromUtf8("label"))
        self.horizontalLayout.addWidget(self.label)
        self.lineEdit = QtGui.QLineEdit(self.horizontalLayoutWidget)
        self.lineEdit.setObjectName(_fromUtf8("lineEdit"))
        self.horizontalLayout.addWidget(self.lineEdit)
        self.label_2 = QtGui.QLabel(Dialog)
        self.label_2.setGeometry(QtCore.QRect(0, 10, 251, 20))
        self.label_2.setObjectName(_fromUtf8("label_2"))
        self.label_3 = QtGui.QLabel(Dialog)
        self.label_3.setGeometry(QtCore.QRect(2, 190, 241, 20))
        self.label_3.setObjectName(_fromUtf8("label_3"))

        self.retranslateUi(Dialog)
        QtCore.QObject.connect(self.buttonBox,
                               QtCore.SIGNAL(_fromUtf8("accepted()")),
                               Dialog.accept)
        QtCore.QObject.connect(self.buttonBox,
                               QtCore.SIGNAL(_fromUtf8("rejected()")),
                               Dialog.reject)
        QtCore.QObject.connect(
                self.lineEdit,
                QtCore.SIGNAL(_fromUtf8("textChanged(QString)")),
                Dialog.say_hello)
        QtCore.QMetaObject.connectSlotsByName(Dialog)

    def retranslateUi(self, Dialog):
        Dialog.setWindowTitle(_translate("Dialog", "Dialog", None))
        self.label.setText(_translate("Dialog", "Name:", None))
        self.label_2.setText(_translate(
                             "Dialog",
                             "<html><head/><body><p align="center">" +
                             "Please enter your name</p></body></html>",
                             None))
        self.label_3.setText(_translate(
                             "Dialog",
                             "<html><head/><body><p align="center">" +
                             "[Where our message will appear]" +
                             "</p></body></html>",
                             None))

It is here that you can learn a lot about how PyQt works, even though some statements seem a bit baroque, like the binding of the textChanged signal to our custom slot:

QtCore.QObject.connect(
        self.lineEdit,
        QtCore.SIGNAL(_fromUtf8("textChanged(QString)")),
        Dialog.say_hello)

If you would write this yourself, you would probably prefer:

self.lineEdit.textChanged[str].connect(Dialog.say_hello)

(Incidentally, the text between the square brackets in textChanged[str] indicates that a single argument of this type is passed to the slot.)

Bringing it all together

Having generated our form class, we can now create a base class that will use this class:

import sys
from PyQt4 import QtGui
from relentless_dialog import Ui_Dialog

class MyDialog(QtGui.QDialog):
    def __init__(self):
        super(MyDialog, self).__init__()
        self.ui = Ui_Dialog()
        self.ui.setupUi(self)
        self.show()

    def say_hello(self, user_text):
        text = "Hello there, {0}!".format(user_text)
        self.ui.label_3.setText(text)

def main():
    app = QtGui.QApplication(sys.argv)
    dialog = MyDialog()
    app.exec_()

if __name__ == "__main__":
    main()

When we are passing self to self.ui.setupUi, we are passing the widget (a QDialog) in which the user interface will be created.

We implement our custom slot by defining a method say_hello that takes a single argument (the user-provided text from the line editor). We craft a witty sentence with it, and set it as the label text.

As mentioned before, we could also dynamically create the GUI directly from the .ui file:

import sys
from PyQt4 import QtGui
from PyQt4 import uic

class MyDialog(QtGui.QDialog):
    def __init__(self):
        super(MyDialog, self).__init__()
        dialog = uic.loadUi("relentless_dialog.ui", self)
        dialog.show()

# etc

Running the application

Running this will show the following (drumroll):

Running the application.
Running the application.

How to use Groovy’s CliBuilder

I write quite some scripts at work, either for myself or for my team. I used to write these scripts in Bash, but since we are using Windows at work, it made sense to switch to a more general scripting language. Because we’re a Java shop, Groovy seemed like a natural choice. Now, it’s often very convenient to have named command-line arguments (e.g. tool --output out.ext --input in.ext) as opposed to positional arguments (e.g. tool out.ext in.ext): named arguments allow for putting the arguments in any order you want, and you can leave out optional arguments and use the defaults. Groovy’s CliBuilder makes things especially easy.

A sample program

weather.groovy

@Grab(group='commons-cli', module='commons-cli', version='1.4')

class Main {
    static main(args) {
        CommandLineInterface cli = CommandLineInterface.INSTANCE
        cli.parse(args)
    }
}

enum CommandLineInterface {
    INSTANCE

    CliBuilder cliBuilder

    CommandLineInterface() {
        cliBuilder = new CliBuilder(
                usage: 'weather [<options>]',
                header: 'Options:',
                footer: 'And here we put footer text.'
        )
        // set the amount of columns the usage message will be wide
        cliBuilder.width = 80  // default is 74
        cliBuilder.with {
            h longOpt: 'help', 'Print this help text and exit.'
            n(longOpt: 'max-count', args: 1, argName: 'number',
                    'Limit the number of days shown')
            '1' longOpt: 'one', "Show today's forecast"
            '2' longOpt: 'two', "Show today's and tomorrow's forecast"
            _(longOpt: 'url', args: 1, argName: 'URL',
                    'Use given URL to query for weather.')
            D(args: 2, valueSeparator: '=', argName: 'property=value',
                    'Use value for given property.')
        }
    }

    void parse(args) {
        OptionAccessor options = cliBuilder.parse(args)

        if (!options) {
            System.err << 'Error while parsing command-line options.\n'
            System.exit 1
        }

        if (options.h) {
            cliBuilder.usage()
            System.exit 0
        }

        if (options.n) {
            // handle user's input
        }
        if (options.'1') {
            // show weather for one day
        }
        if (options.url) {
            URI uri = new URI(options.url)
        }
        if (options.Ds) {
            assert options.Ds.class == ArrayList.class
        }
    }
}

When invoked with groovy weather.groovy --help, the program will output:

usage: weather []
Options:
 -1,--one                  Show today's forecast
 -2,--two                  Show today's and tomorrow's forecast
 -D        Use value for given property.
 -h,--help                 Print this help text and exit.
 -n,--max-count    Limit the number of days shown
    --url             Use given URL to query for weather.
And here can we put footer text.

You can specify short and long names: cliBuilder.n specifies that the program will accept an argument -n. To also specify a long name, we add longOpt: max-count. To use only a long name, we can replace the short name by _: cliBuilder._(longName: 'wow-such-long'). Note that cliBuilder._ can be reused several times (in contrast to other short names).

The number of arguments can be indicated by using the args parameter. If you use more than one parameter, a valueSeparator must be indicated, which is used to split up the argument. I found that the number assigned to args is not very strict: when called with argument -Dmyprop, the following is valid.

assert options.Ds == ['myprop']

Note how we append an s to the option name (options.Ds) to get a list of properties and values. A normal invocation groovy weather.groovy -Dpancakes=yesplease would result in:

assert options.Ds == ['pancakes', 'yesplease']

When required: true is set, the program will shout at you if the argument is not used. For example, if we specify cliBuilder.q(required: true) but fail to provide the -q argument on the command line, the program will exit with error: Missing required option: q.

One thing to remember is that, if you want to use digits as argument names, you have to stringify them first by putting quotation marks around them: cliBuilder.'1'.

More on CliBuilder

Check the Groovy documentation on CliBuilder for more .

Dependencies

Groovy’s CliBuilder depends on cli-commons:

build.gradle

dependencies {
    compile group: 'commons-cli', name: 'commons-cli', version: '1.4'
}

Using Groovy’s AntBuilder to zip and unzip files

Suppose we need to zip a bunch of Python files (without their compiled counterparts *.pyc). The next (admittedly contrived) example shows how we could go about doing something like this in Groovy:

File srcDir = new File('/home/neftas/Downloads/scripts/')
File destDir = new File(srcDir, 'antexample')
File zippedFile = new File(srcDir, 'antzipped.zip')

def allPythonFilenames = srcDir.list().sort().findAll { it.endsWith('.py') }
assert ['a.py', 'b.py', 'c.py'] == allPythonFilenames

def ant = new AntBuilder()

ant.with {
    echo 'begin zipping and unzipping'
    zip(destfile: zippedFile, basedir: srcDir, includes: '*.py')
    mkdir(dir: destDir)
    unzip(src: zippedFile, dest: destDir)
    echo 'done zipping and unzipping'
}

// zip file is created
assert zippedFile.exists()
// contents of zip file should match content of source directory
def commands = ['bash', '-c', "zipinfo -1 ${zippedFile.name}"]
def process = commands.execute(null, srcDir)
def contentsOfZip = process.text.split(System.lineSeparator)
assert contentsOfZip.sort() == allPythonFilenames

// all files should be unpacked from the zip into the right directory
assert destDir.list().sort() == allPythonFilenames

ant.with {
    echo 'deleting created files'
    // notice nested Ant task
    delete(includeEmptyDirs: true) {
        fileset(dir: destDir)
        fileset(file: zippedFile)
    }
    echo 'deleted created files'
}

We can use all the power of Ant in our Groovy scripts. Ant task names map to Groovy methods (see the zip method above) and the attributes are passed as maps to these methods. What’s even better, there is no need to stringify the arguments of these attributes (as is required in Ant’s XML), but we can directly use the correct datatypes (e.g. in the key-value pair destfile: zippedFile, zippedFile is of type java.util.File). All attributes of Ant tasks are available to use with AntBuilder (see Ant manual for zip task).

To use nested Ant tasks, we create closures, so that Ant’s:

<delete includeEmptyDirs="true">
    <fileset dir="lib">
    <fileset file="file.ext">
</delete>

results in the following Groovy code:

ant.delete(includeEmptyDirs: true) {
    fileset(dir: 'lib')
    fileset(file: 'file.ext')
}

Invoking from Gradle

AntBuilder is also available in Gradle. This will come in handy when you don’t know the "Gradle way" of doing something, but do know how to tackle it in Ant.

Dependencies

Groovy (and Gradle) come with a bundled version of AntBuilder, so you should be good shape when creating a Groovy script, but if you want to use AntBuilder in a project, you should add ant and ant-launcher to your build.gradle:

dependencies {
    compile group: 'ant', name: 'ant',  version: '1.7.0'
    compile group: 'ant', name: 'ant-launcher',  version: '1.6.5'
}