Printable View
Hi everyone
Is there a way to retrieve a word at mouse current position regardless of the window? Maybe the a question is a little of confuse, so I describe it in detail. Whenever my applications is running and a user is moving a mouse to particular window that has some texts, I would like my application to pick a word (just a word not all text) at the current mouse position. Can this be done? Thanks for any help.
Well, I have to say it is not an easy work to do.
Typically, you may use hook to capture some call of GDI API function such as TextOut and TextOutA, and then you may modify the image of the functions memory and embed your own assemble code....
This question has been asked a few times, or at least similar questions.
First, are you saying you want to do this while the mouse is moving? That is, not from a mouse click?
Look at the ::GetMessagePos function; you can use that to get the mouse position when a message is being processed. Then you can use ChildWindowFromPoint.
Then it depends on what is there. If I can guess what it is you are doing, then the window will always be one or a few specific windows. Also, my guess is that you want to do this only for edit and/or rich edit controls. In which case, you can get the word by using the location. There are a lot of details, but hopefully that is the general idea.
Hi, Sam Hobbs,
yes, yours is an easy way. But by it, you will only retrieve the title of a certain window such eidtbox or some controls. However, you cann't retrieve words in other cases such as in a webpage, a word process file, a pdf file.....
I doesn't know what is megaxoom's intent. If you only want to get some words from a editbox etc. Sam Hobbs gave you a better way.
Sorry for not describe in more detail, I would like to pick word from any application like : MS Word, Internet Explorer, notepad, etc.
Then you should capture the GDI API although there is still another way that tracking the DC of the windows. but I think the former one does better considering the compatibility. after you capture the API, you may imbed your own assemble codes into the momery of API, or you can change the import address table to run your custom codes.
Hi Olin
Thanks for your help, but could you provide some information where can I start to do this? Thanks.
as olin said, capture the GDI API will work fine. After injecting ur code, each time that API is called, u'll get the start possition of the string, compare it with cursor's possition then u'll get the underlying text. See GetTextExtentPoint32, GetTextAlign for detail
The idea of intercepting 'textout' type calls, and storing them is a bogus solution. It just never will work for a variety of reasons.
1) You still won't know if the text is still there when the mouse is over it. It may already have been painted over by something else.
An easy sample of this would be a tooltip. You may be able to grab the textout that made the tooltip display itself, but it would require a lot of additional work to know that the tooltip has disapeared.
3) Several windows can paint text at identical locations. (an app with several levels of dialogs is a simple example). As a result you would also need to be aware of focus.
4) Text can move around without calling any of the text type functions. Scrolling the window or blitting part of the window to another location is a possibility
5) Many programs (and IE is among them) use off-screen buffers (often this is a MemDC, but not necessarily) to prepare the painting, and then do a single blit from this off-screen buffer to the video memory, thus avoiding flicker.
It's impossble to track text in this type programs.
6) It'll require a HUGE amount of memory to store all the strings that have been painted, and there is no (easy) way to know which strings have become obsolete.
There are possibly a whole bunch of other reasons why capturing and storing GDI calls is just never going to work, the above are just the most obvious ones. This technique is not only going to be incredibly hard to even remotely achieve, it'll never work 'right'.
--
The best possible 'easy' solution is indeed the one Sam Hobbs gave.
Identify the type of window at the cursor-location, and then have a set of Windowtype-specific routines to calculate what text is under the cursor.
Doing this for the the common controls (static, edit, list, tree, combo...) should be easy enough (although quite a bit of work), but other windows are going to be problematic if not impossible as there is no API available for them to do what you want.
Another solution (and I know of a program that does it this way) would involve grabbing a bitmap of the cursor-location, and apply an OCR-type algo on it to figure out what the word is. It works, and works always, but making an OCR algo is going to be a pain...
Ah sigh... Textmode DOS did have it perks in this area ;-)
Assuming that OCR requires a lot of processing, using it in realtime as the mouse scampers around is likely to be impossible also.
Well yes, you would think so, but a picture grabbed from screen is somewhat easier than a document scanned through a scanner...
Text is always perfectly horizontal (well... most of the time) and it's always locked into fixed pixel positions, and you can match charactares to fixed pixel patterns. I'm not saying it's easy (hence the 'pain' I was referring to), but aparently it can be done fast enough to be usable.
This program I was referring has it's word capturing lagging behind somewhat. It won't start grabbing the word until you held the mouse at a specific place for half a second. And it seems to be able to capture the word near instantaneously (on a 2.3Ghz machine, I'll give them that :-)) if it's a 'normal' size (8 to 14pt) and a common font (courier, arial, MS Sansserif, MS Serif, system, terminal...). The kinds of fonts you'd see on you screen 99% of the time.
On extremely large or 'novelty' fonts, processing time can increase to about a second or so.
Still not bad if you ask me...
However this question says that the word should be chosen as the mouse moves, which implies to me that it should be capable of working multiple times per second. I am assuming that is a requirement. If the problem were clarified then perhaps it is reasonable for the solution to require a couple of seconds. However it is probably an insignificant issue in this situation.
There're many ways to know if the text is still there, even if u want, the tooltip won't have the chance to show. Anyway, I think there're not many tooltips which like to overlap its text.Quote:
1) You still won't know if the text is still there when the mouse is over it. It may already have been painted over by something else.
An easy sample of this would be a tooltip. You may be able to grab the textout that made the tooltip display itself, but it would require a lot of additional work to know that the tooltip has disapeared.
There's no need to aware of focus, u get the necessary window just by one or two functions. If there're several windows at an identical locations, only the topmost can receive the invalidate region, that means, triggers 'textout'.Quote:
3) Several windows can paint text at identical locations. (an app with several levels of dialogs is a simple example). As a result you would also need to be aware of focus.
Text always has a relative co-ordinate, called logical co-ord. And u can convert this to screen co-ord very easily.Quote:
4) Text can move around without calling any of the text type functions. Scrolling the window or blitting part of the window to another location is a possibility
It's possible coz before blitting to the video, it must 'textout' its off-screen buffer first.Quote:
5) Many programs (and IE is among them) use off-screen buffers (often this is a MemDC, but not necessarily) to prepare the painting, and then do a single blit from this off-screen buffer to the video memory, thus avoiding flicker.
It's impossble to track text in this type programs.
U don't have to store any string, GDI functions did that. Really, if u draw a bunch of text, it will be divided into smaller sections, so no 'textout' function draws more than one line.Quote:
6) It'll require a HUGE amount of memory to store all the strings that have been painted, and there is no (easy) way to know which strings have become obsolete.
Of coz, this solution doesn't work in all cases. But it will work in almost case. Beside, u don't need to care about the processing time, the OCR algo, not much about the font... I think OCR solution is a good solution too, but it won't work in all case, either
I have not heard of logical coordinates, especially not that can be converted to screen coordinates. Do you mean client coordinates?Quote:
Originally posted by sephiroth2m
Text always has a relative co-ordinate, called logical co-ord. And u can convert this to screen co-ord very easily.
No, Sam, it's logical coordinates, that almost GDI functions use. Screen/window/client coordinates are device coordinates (that have origin at the top-lefft and unit = pixel). U use functions such as LPtoDP, ClientToScreen... to convert between them
A tooltip is just an easy example, but I can think up many more cases where different texts would just be overwritten one on top of the other.Quote:
There're many ways to know if the text is still there, even if u want, the tooltip won't have the chance to show. Anyway, I think there're not many tooltips which like to overlap its text.
How about a progress bar with a text-control that just overwrites the previous progress step. It would be very complicated to keep track of which texts are no longer applicable.
A window does not need to be topmost to be able to get a textout (or even have any of it's area visible for that matter). Note that not all programs are optimized to ONLY repaint the invalidated regions, but just repaint everything the clipping Windows takes care of making sure those programs don't paint on another windows' canvas.Quote:
There's no need to aware of focus, u get the necessary window just by one or two functions. If there're several windows at an identical locations, only the topmost can receive the invalidate region, that means, triggers 'textout'.
Many programs also don't just paint as a response to WM_PAINT messages, but do painting at the moment an event occurs, and rely on clipping to take care of everything.
At the very least, you would need to track every textout, and pair it up with the window it's painted on. And even then things get hairy as there's still transparency to deal with.
You don't understand me I'm afraid. I'm not talking about using the scrollbars to change the offset of the window inside it's 'container'.Quote:
Text always has a relative co-ordinate, called logical co-ord. And u can convert this to screen co-ord very easily.
There are DC functions to scroll the contents of a window without changing any of the coordinates.
It's essentially the same as doing a blit from videomemory to another location in videomemory. However most accelerated Videocards have explicit support for this, making a scroll faster than a blit. See MSDN for ScrollDC()
True, I wasn't trying to imply that it didn't need a textout.Quote:
It's possible coz before blitting to the video, it must 'textout' its off-screen buffer first.
But it makes things an order of magnitude more complicated. I was just trying to show that ONLY tracking all the textouts (and any of the other forms of drawing text) isn't going to be enough.
GDI functions took a string, converted it to a bunch of pixels rendered to screen. Once that's done, GDI no longer is aware of the strings it painted. If you need to be able to analyze what string is at the cursor location, the program is going to need to store these strings itself before/after actually calling the GDI function to do the drawing.Quote:
U don't have to store any string, GDI functions did that. Really, if u draw a bunch of text, it will be divided into smaller sections, so no 'textout' function draws more than one line.
I think you fail to see the complexity of how everything ends up on the screen.Quote:
Of coz, this solution doesn't work in all cases. But it will work in almost case. Beside, u don't need to care about the processing time, the OCR algo, not much about the font... I think OCR solution is a good solution too, but it won't work in all case, either
The OCR solution is NOT all solving either, I'm aware of that. Reviewing my post I seem to have implied that it does, but indeed it doesn't. But the solution of trying to track down, analyze and store all the GDI calls is in my eyes a near impossible task.
If it indeed were ONLY a matter of tracking the TextOuts it were somewhat feasible, but there's a whole other things to be converned about.
1) You need to track all forms of outputting text.
TextOut, TextOutEx, DrawText, ...
You would need to store the coordinates, the actual string, fonttype, size, alignment, orientation
And you would need to be able to reverse a coordinate that lies within the painted area into the word of the entire string.
2) You would have to be aware of what window the text was output to.
3) You would need some form of 'garbage collection' to remove the 'stale' data. Strings that got overwritten with something else, windows that got destroyed etc.
4) You would need to handle blits and scrolls and work out which parts of previously painted text they are moving (or copied) to another location.
This moving/copying around of portions of videomemory happens A LOT !!! It's not something that's used by only a small amount of programs.
It's the single thing that would make this entire scheme fall in the water.
5) You would also need to be aware of clipping regions active when either the textout or blit/scrolling was called.
6) Going into somewhat more extreme, you would also need to handle transparency, but I'll agree to the fact that this particular feature is somewhat less frequently used.
This is confusing, since ClientToScreen is used to convert Client coordinates to Screen coordinates, right? So I can't be sure how relevant the issue is. I am being vague because it is not worth the time to refer back to what has been said to refresh my memory.Quote:
Originally posted by sephiroth2m
U use functions such as LPtoDP, ClientToScreen... to convert between them
Since the subject is text, logical coordinates have little or not relevance. In nearly all relevant situations, the text will be drawn using text mode, which is one-for-one pixels, or however it is said. Regardlous of how it is said, the mapping mode is rarely or never a problem relevant in this context.
@OReubens: We seem to misunderstand each other. Actually, I think there're not so much things to do with this method (as u said). Consider the folowing:
-Get the window under cursor (by WindowFromPoint, RealChildWindowFromPoint...)
-Invalidate the window that covers the cursor
-If there're some text, the window will call 'textout'
-Maybe many 'textout's will be called, we'll capture all, then verify which is under the cursor
-Get the text in that 'textout'
Again, I don't think this solution works in all cases. But it's quite simply and 'acceptable'
@Sam: In the case of comparing the mouse's co-ord (screen co-ord) with a text's co-ord (logical co-ord), i think those functions are necessary. Anyway, some word-processing apps could change their mapping mode so they could TextOut basing on lines, not on pixels.
I don't think invalidating the window is a good idea. You are needlessly hogging the system.
the solution for this seems to be highly window specific. Probably you will have a handler for each known window class that you need to trap, static, edit, combo, dialog etc...
Yes, certainly. However that is unusual; for example, is that not possible using Word without using tool or component? TextDraw or something like that? So I did not say it does not happen, I am just saying it is unusual. In fact, I think Word does not, as I said, do it; it is the tool or component that does.Quote:
Originally posted by sephiroth2m
some word-processing apps could change their mapping mode so they could TextOut basing on lines, not on pixels.
Microsoft Active Accesibility can give you the text under mouse cursor, but it wont tell you the exact word.
so if we get the text (actually the text paragraph who has same IAccesible), how to calculate what word is behind the cursor ??