Identifying known image display by other app (screen scraping, not web based)

I need to develop a program that can identify a known rectangle image/pattern being displayed on the screen in a fixed location. I also need this program to identify text being displayed on the screen in a fixed location.

There are about seventy-five (75) such images. I know what they all look like, and can create a pool of bitmap images to match against, or other similar method so the program has something to match against.

I know such a program is possible, I've seen others that perform this very ability. Unfortunately the program displaying these images/patterns and text that I need to identify has no API to interface with it, and is a Windows application. I think what I'm trying to do is called screen scraping, but unfortunately all I have been able to find on screen scraping is either for getting information from websites or text terminals.

If anyone could give me some leads on how to start on this, I would greatly appreciate it. I'm not sure if I need to have the program grab a full screen screenshot and start analyzing that, or if it's possible to have the program grab a screenshot of the specific window that's open (like ALT+PRINTSCREEN) so it can read the information even if the target window is being hidden or minimized.

I have a lot of programming experience, but none of it is with Windows graphical applications. I'm not sure how to get such screenshots, and once I have that, how to search for a known image/pattern within the screenshot.

I don't mind doing this in C++ or C#. Using the .NET framework is fine.

License issues aside, have you considered hooking the target application's drawing function(s) (or even high-level logic functions) as a means of extracting information?

OCR is a little overkill for this type of problem, IMO.

Unfortunately there are license issues involved that I can't violate. Under the license, I can't hook into the target application at all, but under the license I am still allowed to have a program look at it's output on the screen.

Years ago, I helped develop an automation system that tested a wrapper around direct x api's. As such we tested on about 80 machines each with a different video card and multiple Windows OS's (Win9x -> WinXP) per machine.

We built up the automation to include around 3000 api tests and originally we had about 100 manual tests there were performed by several manual testers which amounted to a screen that would display an image and the tester would choose pass or fail based on if they liked what they saw.

This manual testing was extremely tedious and time consuming. If I recall correctly, a complete pass across all machines/platforms took several testers about 3 weeks to complete.

Our automation system had become highly developed and we could with a minimal of monitoring essentially hit the GO button, and the machines would automatically get reimaged (for a new OS), the tests would be executed, the next OS would be installed and so on. Running 24/7, the whole pass would take 3 days.

The manual test time was killing us, so we needed a way to reduce this part of the testing.

We all had had experience with other forms of automated visual testing. A popular approach involved taking screen shots and saving the images to a database and using them for the compares.

We rejected this approach because creating screen shots for the differences between resolutions, color depth, video cards and drivers would not be very reliable and managing all those images would be a maintenance nightmare.

We needed an approach that was more self contained and performed more of a fuzzy match. We found that different video cards (and even different video drivers) produced different display outputs that wasn't necessarily visible to the naked eye.

Lastly we want a system that could handle the offset of a few pixels. In other words, the system should be able to perform a compare even if the image was over to the left (right, up or down) a slight bit.

What we ended up doing was something that is conceptually very simple. Given a screen region we simple totalled up the pixels within the region and calculated the average color for the pixels.

Then within the test code we allowed the tester to write a series of regions (with the corresponding color average) to adequately test the complete image.

For example, consider a 1" solid yellow circle centered on a 4" x 4" black background. For this simple example there may be a total of 5 regions: 1 over the circle and the other 4 regions capturing the area around the circle.

The region over the yellow circle would yield and average value that is close to yellow (although this region contains black in the corners). The region surrounding the circles should all be virtually black. If, for example, one of the black regions show a color value that isn't close to black, you know that either the yellow circle is in the wrong position or some other issue has appeared on the screen.

We developed a tool that would help the test developer define the regions and compute the color average. The tester would enter a line of code into the test code that included the region boundaries, the color, the offset, and acceptable deviation.

The approach is surprisingly simple, but you can perform some complex image comparisons by simply building up the amount of required regions. You can even detect a limited amount of text.

What I would envision for your situations is that you can build up a database of regions that would identify one of your 75 screens and during the comparison process, just run through your region collection to detect which screen you have.

A system like this takes quite a bit of initial work to get going, but the payoff can be rewarding.