HW1 - Image Retrieval using Visual and Textual Features (due Feb 16, 11:59pm)
In this homework you will explore simple image and text features in a retrieval scenario. You can program in the language of your choice (matlab and/or python are encouraged).
Download this shopping data collection shopping.tar.gz.
This data contains real shopping images from the web depicting 2
categories (bags and shoes) and text descriptions associated
with each image. The "images" directory contains your database of web images and descriptions, while
"queryimages" contains example query image-description pairs.
Part 1 - Image Feature Based Retrieval (25 pts)
- Write a function that retrieves similar images based on grayscale tiny-image descriptors
Your code should take as input the filename of a query image and directory location of the
- Within your code you should first compute grayscale tiny-image
descriptors (image converted to grayscale and resized to icon size of 32x32
pixels) for each image in the database and for the query image. Then you should
print out the filenames of the 10 most similar images in the database using SSD
on your image descriptors.
Part 2 - Text Feature Based Retrieval (25 pts)
- Write a function that retrieves similar images based on word vector
descriptors of their associated textual descriptions.
Your code should take
as input the filename of a query image description and directory location of the database.
- Within your code you should first compute a lexicon for the database
descriptions (you should try variations on lexicons and select one that
produces good results, e.g. all words observed in any description, commonly
occurring words, with stop words removed, and so on). Next you should compute
the word vector descriptors for the database descriptions and query image
description. Then you should print out the filenames of the 10 most similar images
in the database using SSD on your text descriptors.
Part 3 - Combined Text & Image based Retrieval (20 pts)
- Write a function that retrieves images using combined image and text
similarity. Your code
should take as input the filename of a query image, query description, and directory location of the
- Within your code you should compute the same descriptors as in Parts 1 and
2, calculate SSD based on image descriptors and text descriptors, then combine
the two SSD calculations in a weighted sum (alpha*ImageSSD +
(1-alpha)*TextSSD). You should experiment with possible values for alpha.
Part 4 - Freestyle (30 pts)
- Implement several extensions of your choice.
- For example: try additional image and/or text descriptors, different
similarity computations, incorporate PageRank style importance estimations into
your experiments, or download some new query images from the web and see what
What To Turn In
Submit by email to firstname.lastname@example.org:
- Your commented code.
- A ReadMe describing your code and how to run it.
- A comprehensive write-up that includes:
- Visualizations for Parts 1, 2, and 3 showing the image-description pairs retrieved for
example images in the "queryimages" directory.
- Description of the lexicons you tried in Part 2 and what lexicon you selected for best results.
- Description and examples of how performance varied as a function of alpha
in Part 3.
- Description of where the algorithms worked, where they didn't, and why you
think that might be the case. You might want to look past the 10 closest images
to get a better sense of how the various algorithms are working.
- Description and results for the extensions you implemented in Part 4.
Tips (if using matlab)
- The most important command in matlab is "help", "help images" will list
useful image related functions, and you can get help for particular functions
by typing help function name, e.g. "help imagesc" or "help montage".
- The "save" and "load" commands are quite useful to save computation time while debugging code.
- One useful trick I use all the time is to write a simple for loop in matlab that creates a
webpage showing results. Here is an example.