HW2 Mining Image Labels from Web Text Descriptions for Classification
Due Oct 10, 11:59pm
In this homework you will train classifiers for color based visual attributes.
Training images will automatically be labeled by mining the text descriptions associated
with web shopping imges. You will then use these classifiers to retrieve images
displaying each attribute from a collection of testing images.
We will again use shopping images, this time the bag portion of the dataset --
Part 1 - Mining Image Labels from Descriptions
We will be training attribute classifiers for 5 color terms ("black",
"brown", "red", "silver", and "gold"). In this part of the homework you will
automatically collect training and testing images from the bag dataset by
utilizing their existing associated text descriptions.
- Write code that can determine for an attribute term, whether that attribute
term is present in an image description. You should implement this in matlab using
the string processing functionality (help strfun gives a list of useful string
processing functions). Note: your code should match both upper and lower case
versions of each attribute term and should include processing to remove
any punctuation (an example function to do this is
- The set of images whose descriptions contain exactly one of the
attribute terms will form your training set. The rest of the images (with
descriptions containing multiple or no attribute terms) will form your testing
Part 2 - Computing Image Descriptors
- Compute a hue-saturation-value color histogram descriptor for each image in your training and
testing sets, using 10 bins for each color dimension (h,s, and v).
This will be the image descriptor used in the remainder of the homework. Remember
to normalize your histograms so that they sum to 1. Note, you can remove images that are
grayscale from consideration since they won't have valid color histograms.
Part 3 - Training Classifiers
In this part of the homework you will use RBF kernel SVMs to train attribute classifiers that
recognize images displaying a color-based visual characteristic. Positive examples for training an
attribute, e.g. "black", will consist of those images in your *training set* that have the attribute
in their text description. Negative examples will be the rest of the images in your *training set*.
- You should use the Matlab LibSVM package located here as
your SVM implementation (description of LibSVM here).
Make sure to add the location of your LibSVM
installation to your path using the "addpath" command. In this package, SVM training is
implemented as a function called svmtrain, while SVM testing is called svmpredict. Note, you should call
these functions directly from within Matlab (the README file within the libsvm-3.12/matlab directory provides installation and
- Because your SVM will be sensitive to choice of parameter values, split
your training set into two parts: 70% as a temporary training set, and 30% as a
tuning set. Use the tuning set to select good values for the SVM parameters -- C and g.
To do this you can simply search over a range
of reasonable values for C and g. Here you will train your SVM on the temporary training set and evaluate
accuracy on the tuning set. The best parameter settings for each attribute will
be those that produce the best accuracy on the tuning set. Report parameter settings
and best tuning accuracies in your write-up.
- Once you have found good parameter values (for each attribute), use the
entire training set to train final SVM models (for each
attribute). Use the '-b 1' option to get probability values out of your SVM.
Part 4 - Classifying/Retrieving Images without Attribute Annotations
Here we will retrieve images displaying visual attributes from your testing
set (collected in Part 1).
- Classify each image in your testing set using the '-b
1' option to get a probability value.
- For each attribute, rank the test images according to their probability.
Create a web page for each attribute showing the most likely 200 images in
ranked order. This ranking should make sense if you've implemented everything
- Compute precision@k curves for each attribute for k=1...200. Note since you don't have
labels for the test set, you will have to calculate these precision values by looking at the results
and determining whether each ranked image displays the attribute or not.
What to turn in
Hand in via email to email@example.com:
- A write-up including a description of what you implemented, your best parameter
values for each attribute and tuning set accuracies, and precision@k curves for each
- 5 web pages showing top 200 results for each attribute -- note you don't need to submit
the images for these pages, just assume they are located in a directory, "bags" in the same
location as your web pages. Note you can also post these online if you wish, but please submit
the webpages with your submission.
- Commented code.
- ReadMe documenting code.
Responses to Q's from students (updated 10/7)
- LibSVM functions can be called directly from within Matlab.
- Instructions for installation and usage of LibSVM functions are
provided in the libsvm-3.12/matlab/README file. Make sure you are using the
correct svmtrain function by adding the libSVM directory to your path using the "addpath" function.
- You should train 1 SVM for each attribute (5 models total). Classification
of a test instance should be computed as the attribute classifier with highest
- Specific training usage help: training_instance_matrix - should be a matrix
containing your computed color histograms for training images,
training_label_vector - should be a vector of training instance labels, e.g.
1/-1, libsvm_options - should specify libsvm parameters such as values for C, g, b.