Why Visual Question Answering?

Visual Question Answering describes a form of Computer Vision problem where you give the method an image, and a question, and it gives you the answer.  Actually, that’s kind of true about a lot of problems in Computer Vision, but the difference in VQA is that the question isn’t determined beforehand, you can make it up on the spot.  That means that the VQA method has to be ready to interpret your question (as text), and the image, and to be able to answer it, despite the fact that it has no idea what you’re going to ask.

This is a big change for Computer Vision, where we typically determine the question, then collect the training data, then develop the method, and only then apply it.  This means that the question is asked at least 6 months before it’s answered.

In VQA, in contrast, the question is not known until run-time, and it is answered as soon as it is asked.  This means that a VQA method has to be ready to answer any possible question about an image.  It thus has to be able to understand the question, understand what’s in the image, then calculate, and formulate an answer.

Achieving this is very close to achieving real Artificial Intelligence.  The Turing test of AI requires a method to answer a set of text questions, and this is harder, because it needs to understand images too.

Fortunately, most of the questions in the standard datasets are relatively achievable.  One of the questions associated with the image above, for example, is “Does the lunch truck sell chicken quesadillas?”.

Deriving a method which can answer a question like this is very challenging, but we’re making some great steps.  We have led most of the benchmarks in VQA in 2016/7 and just won the international VQA challenge for 2107.  We have also published a host of papers illustrating how external information can be exploited, and more difficult questions can be answered.

Number 1 in VQA

The results have just been announced and we won the top spot in the latest Visual Question Answering challenge VQA2.  The results are announced each year at the VQA Workshop at CVPR.   It’s a very hotly contested challenge, with teams entering from around the world.

The credit goes to Damien Teney (ACVT) and Peter Anderson (ANU & ACRV) primarily, but the supporting cast includes people from Microsoft, Stanford, and Adelaide of course.

It’s a really interesting challenge because it’s the primary driver for Computer Vision beginning to tackle problems that approach AI.

One of the questions associated with the image above, by the way, is ‘Are the Dodgers playing?’.

Number one in CityScapes

Cityscapes is a dataset which is designed to test semantic segmentation methods.  These are methods that can determine what type of object a pixel belongs to, for every pixel in an image.

The reason that this has become such a hotly contested challenge is that it’s a crucial capability for driverless cars.  Driveless cars need to be able to ‘see’ where the road is, to identify pedestrians and other vehicles, trees and street signs, and Cityscapes tests exactly these capabilities.

The team that won was Zifeng Wu, Chunhua Shen, and me.  This was achieved using our new Deep Learning architecture.  This is the same architecture that was behind our entry into the Scene Parsing challenge in ImageNet 2017, which came 2nd.  Our model had the best single-model performance, but was just edged out by an ensemble of weaker methods.

Number 2

This year we were number 2 in the ImageNet Scene Parsing Challenge.   ImageNet is one of, it not the, most hotly contested challenge in Computer Vision each year.  It attracts entries from most of the major groups in the field, both commercial and academic.

Second is a great result in this incredibly competative challenge, particularly as it was achieved by developing a new architecture which outperforms the very deep networks.  That is, although our ensemble of networks was outperformed (slightly) by the team that won, our single model performance was the best.  It is this single model performance which gives the best indication of the capacity of the architecture, which is why we’re so enthusiastic about the result.

The architecture we used was wide, rather than deep.  This has a number of advantages, most of which reflect the fact that there is better parallelism in a wide network than a deep one.

15 ACVT CVPR Papers Accepted

We’ve had 15 papers accepted to the 2016 IEEE Conference on Computer Vision and Pattern Recognition, which is a record for us, and I suspect for any group in Australia, ever. CVPR is the best conference in Computer Vision.  It receives over 2000 full paper submissions which are double blind reviewed for an acceptance rate of less than 30%.

Number 4

The latest ImageNet results are out and we’re number 4 in the world!  That makes us the first non-commercial entry as the Chinese University of Hong Kong team were very closely aligned with a very well funded Chinese company.  We thus placed ahead of Google, Intel, and Tencent.

One of the ImageNet images

This is all the more impressive as we had a fraction of the computing resources of these competitors.

Bring me a spoon

The fact that robots can reliably navigate the halls, and streets, without bumping into the walls has been one of the great successes of robot vision, and particularly Simultaneous Localisation And Mapping.

My version of the challenge ahead, however, is based on the fact that we are still a long way from being able to usefully ask a robot to ‘bring me a spoon’.  Obviously the spoon itself isn’t the key feature, it’s the amount of knowledge you need to have to be able to respond to general commands, given by humans. A 5 year old isn’t allowed to drive a car, weld, or vacuum, all tasks that robots carry out every day.  A 5 year old understands, however, that spoons are found in drawers, which are found in kitchens, and knows how to operate the drawer, and pick up a spoon.  A robot has none of this knowledge.

So we’re still a long way from being able to ask a robot to bring me a spoon, but there are a set of really interesting challenges along the way.  Not bumping into the walls is a good first step, and being able to recognise spoons will help, but the more interesting question is how we reach the point where the robot is able to gather the information required to complete the task, and use that information to come up with a solution.  5 year olds are even able to asks questions about parts of the problem that they don’t understand.

Visual question answering

One of the things I’m most interested in at the moment is Visual Question Answering (VQA).  The idea is that you ask a question, posed in Natural Language, that relates to the content of an image.

Visual question answering is distinct from many problems in Computer Vision because the question to be answered is not determined until run time. In more traditional problems such as segmentation or detection, the single question to be answered by an algorithm is predetermined, and only the image changes.

This means that someone defines the question, including the image set that it relates to.  The researcher then pays a large number of people (often in a third world country) to do thousands of hours of work in hand labelling the images. The community then spends the next few years working on the problem of devising a system to replicate the human answers.  This is not a real-time process.

In visual question answering, in contrast, the form that the question will take is unknown, as is the set of operations required to answer it. In this sense it more closely reflects the challenge of general image interpretation.

VQA typically requires processing both visual information (the image) and textual information (the question and answer). One approach to Vision-to-Language problems, such as VQA and image captioning, which interrelate visual and textual information is based on a direct method pioneered in machine language translation. Rather than building a high-level representation of the text, and re-rendering that high-level understanding into a different language, the direct approach develops an encoding of the input text using a Recurrent Neural Network (RNN) and passes it to another RNN for decoding. This is significant because, although this method does not form a high-level representation of the meaning of the text, it performs well.

So we’re in the position where a method which is incapable of performing any real reasoning about image content is providing the best performance.

This is the issue that I want to address, by developing VQA methods which are actually capable of reasoning about the content of the image.

Positions available

We’re advertising again.  Have a look here for the latest post-doc positions, but we’re about to announce some programming positions too.

Proving performance

Apas is a system for identifying microbial growth on agar plates, such as might be used in a pathology lab or hospital to improve the processing of biological samples from patients.

The ACVT developed the technology behind Apas with LBT Innovations, and Adelaide biomedical company.

Results of a study of the performance of Apas were presented to the Australian Society for Microbiology this month, showing that Apas didn’t miss a single pathogen in over 500 plates.

More details are available here.