Problem: Cancelling messages with CallWndProc

Problem: Cancelling messages with CallWndProc

I’ve created a library in c++ 2008, which uses SetWindowsHookEx and CallWndProc.

I’m attempting to stop some messages from being processed, and can’t seem to get it right.

I’ve trimmed the code down a bit, but basically all this code is in dllmain.

Also, I’m running StartHook and StopHook from a c# app using pinvoke.

I am definitely receiving the messages, which you can check by uncommenting the messagebox code at the bottom (be careful with this though, it will most likely crash your system. Try it in a virtual machine if you have to!)

// dllmain.cpp : Defines the entry point for the DLL application.
#include “stdafx.h”

LRESULT CALLBACK CallWndProc(int nCode,WPARAM wParam,LPARAM lParam);
HHOOK myHook = NULL;
HMODULE myModule = NULL;

extern “C” void _declspec(dllexport) StartHook(){
if (myHook == NULL){
myHook = SetWindowsHookEx(WH_CALLWNDPROC,(HOOKPROC)CallWndProc,myModule,0);
}
}
extern “C” void _declspec(dllexport) StopHook(){
if (myHook != NULL){
UnhookWindowsHookEx(myHook);
}
}

BOOL APIENTRY DllMain( HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved){
switch (ul_reason_for_call) {
case DLL_PROCESS_ATTACH:
myModule = hModule;
break;
case DLL_THREAD_ATTACH:
break;
case DLL_THREAD_DETACH:
break;
case DLL_PROCESS_DETACH:
StopHook();
break;
}
return TRUE;
}

LRESULT CALLBACK CallWndProc(int nCode,WPARAM wParam,LPARAM lParam){
if (nCode message){
case WM_SIZING:
cancelMessage = TRUE;
break;
case WM_SIZE:
cancelMessage = TRUE;
break;
}

if (!cancelMessage){
return CallNextHookEx(myHook,nCode,wParam,lParam);
}else{
return 1; //This doesn’t work
// MessageBox(0,L”Message Cancelled”,L”Cancelled”,MB_OK);
}
}

---

Solution: Cancelling messages with CallWndProc

In support of my supposition:
    http://msdn.microsoft.com/en-us/library/ms644959(VS.85).aspx
    Note that the hook procedures for some types of hooks can only monitor messages. the
    system passes messages to each hook procedure, regardless of whether a particular
    procedure calls CallNextHookEx.

Furthermore, the discussion of that particular kind of hook (WH_CALLWNDPROC) is quite clear as to its nature as a notification tool — the hook does not see the message itself, but rather a copy of the message that *is about to be sent* (WH_CALLWNDPROCRET hooks get to look at the message that *has already been sent.*

Given that, it’s easy to see that you can’t cancel the message itself.  At most you might be able to prevent other hooks in the hookchain from seeing the message.

And note:
     CallWndProc Function
     http://msdn.microsoft.com/en-us/library/ms644975(VS.85).aspx

    The CallWndProc hook procedure can examine the message, but it cannot modify it.
    After the hook procedure returns control to the system, the message is passed to the
    window procedure.

So there’s your answer.  
You can’t modify it and you can’t cancel it because this type of hook does not actually intercept the message.  It is used to get notified when a certain message is about to be sent.

=-=-=-=-=-=-=
It appears that the WH_GETMESSAGE can get a chance to modify the message itself.  In that case, you could try changing MSG.message to some other value (such as a value above WM_USER) or perhaps better, WM_NULL (0).