Problem : Microsoft Full-Text Engine Filter Daemon process at 100%

Problem : Microsoft Full-Text Engine Filter Daemon process at 100%

I am having an issue with my SQL Server since we upgraded it to 2008 (from 2005).

Although the text search works fine the Microsoft Full-Text Engine Filter Daemon process (fdhost.exe) is permanently running and taking 100% of the processor – slowing the machine up.

Details:
I have a database containing a few tables, but the core data is in two tables which are FTS indexed. Those two tables store 15GB of PDF’s in varbinary(max) fields.

I dropped the FTS indexes and associated catalogues, made new files and filegroups in a different location, re-added the FTS indexes, and set the catalogues to be inside the new filegroups; in an effort to allow 2008 to rebuild things without any issues due to the 2005 upgrade.

I didn’t set any tracking on the tables, as only one will get updated, and that will be rare, so a nightly manual update of the index’s/ catalogues will be sufficient.

If I watch the process monitor, I can see where the indexing finishes; as the usage stops going up and down and goes to a static 100%.

The ErrorLog shows the full-text filter daemon starting and stopping without issue.
The SQLFT00000….log shows the indexing completing successfully.

“sp_help_fulltext_catalogs” shows Status 0 – idle.

To get it to stop I have to set the service to manual, stop it and then end the process.
Restarting the service (still in manual) loads the processes into memory but at 0% cpu usage. (CPU runs at 1-2% then).

So I am a bit perplexed why the fdhost, after successfully re-indexing (proven by querying the catalogue, and inspecting the fts index properties) goes to 100% for days!

ErrorLog

2009-06-18 19:37:24.68 spid28s The full-text filter daemon host process has stopped normally. The process will be automatically restarted if necessary.
2009-06-18 19:37:24.81 spid28s A new instance of the full-text filter daemon host process has been successfully started.
2009-06-18 20:13:22.75 spid22s The full-text filter daemon host process has stopped normally. The process will be automatically restarted if necessary.
2009-06-18 20:13:22.86 spid22s A new instance of the full-text filter daemon host process has been successfully started.SQLFT00000….log
2009-06-18 20:18:42.69 spid21s A full-text retry pass of Full population started for table or indexed view ‘XXXXXXXXXXX’. Table or indexed view ID is ‘133575514’. Database ID is ‘6’.
2009-06-18 21:21:40.09 spid26s Informational: Full-text retry pass of Full population completed for table or indexed view ‘XXXXXXXXXXXXX’ (table or indexed view ID ‘133575514’, database ID ‘6’). Number of retry documents processed: 800. Number of documents failed: 0.

Solution : Microsoft Full-Text Engine Filter Daemon process at 100%

Have worked through the issue with Microsoft, and they analysed a memory dump file and surprise – Microsoft are saying there is a problem with Adobes iFilter…

“all the things our FDHost.exe was doing when you encountered the high CPU issue was inside that AcroRdIF.dll. As it’s a 3rd party software, we don’t have the source code of that (even we don’t have symbol), it’s impossible for us to tell you what going wrong inside that and why the AcroRdIF.dll kept taking up CPU. That’s why we need your help to engage adobe side. Please see the attachment for dump analysis attached. (as there are lots of possibility of such kind of issue, without the help from the software provide, we cannot just guess the root cause)”

“Besides that, I have suggested you can try to load FoxIT PDF iFilter instead of Adobe PDF iFilter. Have you tried that? If you have any difficult to find the FoxIT PDF iFilter, you can find it there:
http://www.foxitsoftware.com/pdf/ifilter/

According to MSDN blog, it’s 4 times faster than the Adobe one. ”

I am not really interested into moving to a non-Adobe filter, as we have recommended to the client to make the pdf with acrobat, to ensure compatibility with the filter..

Looks like this is another dead end for now.

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:

Comment: 'Full dump in Hang Mode for FDHOST.EXE_running_on_**********
Symbol search path is: SRV*D:\SymCache*\\gcrcssfs\Products\Symbols\*\\cprsqltx\symbols\*http://symquery001
Executable search path is:
Windows Server 2003 Version 3790 (Service Pack 2) UP Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Jul 20 13:19:47.000 2009 (GMT+8)
System Uptime: 19 days 2:26:02.305
Process Uptime: 0 days 0:00:58.000
...................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(e8c.ea4): Wake debugger - code 80000007 (first/second chance not available)
eax=00001001 ebx=00000000 ecx=00001010 edx=001a8008 esi=000000bc edi=00000000
eip=7c82860c esp=000afa78 ebp=000afae8 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
7c82860c c3              ret
We can see there were only 6 threads running in the FDHost.exe.
0:000> ~*kL
 
.  0  Id: e8c.ea4 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
000afa74 7c827d29 ntdll!KiFastSystemCallRet
000afa78 77e61d1e ntdll!NtWaitForSingleObject+0xc
000afae8 6001856b kernel32!WaitForSingleObjectEx+0xac
000afb04 6001d666 fd!CSrchEvent::Wait+0x22
000afb10 010035f0 fd!CSrchFDService::WaitForShutdown+0xc
000aff7c 01003b47 fdhost!wmain+0x9a4
000affc0 77e6f23b fdhost!__tmainCRTStartup+0x10f
000afff0 00000000 kernel32!BaseProcessStart+0x23
 
   1  Id: e8c.240 Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr
0070fea0 7c827d19 ntdll!KiFastSystemCallRet
0070fea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
0070ff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
0070ffb8 77e6482f ntdll!EtwpEventPump+0x27f
0070ffec 00000000 kernel32!BaseThreadStart+0x34
 
   2  Id: e8c.320 Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr
009eff9c 7c826f69 ntdll!KiFastSystemCallRet
009effa0 7c83d454 ntdll!ZwDelayExecution+0xc
009effb8 77e6482f ntdll!RtlpTimerThread+0x47
009effec 00000000 kernel32!BaseThreadStart+0x34
 
   3  Id: e8c.748 Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr
00a6ff70 7c8277f9 ntdll!KiFastSystemCallRet
00a6ff74 7c839f68 ntdll!NtRemoveIoCompletion+0xc
00a6ffb8 77e6482f ntdll!RtlpWorkerThread+0x3d
00a6ffec 00000000 kernel32!BaseThreadStart+0x34
 
   4  Id: e8c.ebc Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr
00ebfd88 7c827d19 ntdll!KiFastSystemCallRet
00ebfd8c 77e6202c ntdll!NtWaitForMultipleObjects+0xc
00ebfe34 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
00ebfe50 600397f4 kernel32!WaitForMultipleObjects+0x18
00ebfe70 6003ae27 fd!CWBServerBufferManager::WaitForClientOrShutdown+0x29
00ebfeb4 6003ae72 fd!CWBServer::WBWorkItemNoHandler+0x12e
00ebfeec 7c83a857 fd!CWBServer::WBWorkItem+0x18
00ebff44 7c83aa3b ntdll!RtlpWorkerCallout+0x71
00ebff64 7c83aab2 ntdll!RtlpExecuteWorkerRequest+0x4f
00ebff78 7c839f90 ntdll!RtlpApcCallout+0x11
00ebffb8 77e6482f ntdll!RtlpWorkerThread+0x61
00ebffec 00000000 kernel32!BaseThreadStart+0x34
 
   5  Id: e8c.eb8 Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr
00f5ff70 7c8277f9 ntdll!KiFastSystemCallRet
00f5ff74 7c839f68 ntdll!NtRemoveIoCompletion+0xc
00f5ffb8 77e6482f ntdll!RtlpWorkerThread+0x3d
00f5ffec 00000000 kernel32!BaseThreadStart+0x34
 
   6  Id: e8c.2d0 Suspend: 1 Teb: 7ffd7000 Unfrozen
ChildEBP RetAddr
01fbff44 7c827d99 ntdll!KiFastSystemCallRet
01fbff48 77e4e02f ntdll!NtYieldExecution+0xc
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for AcroRdIF.dll -
01fbff4c 10001c2a kernel32!SwitchToThread+0x6
WARNING: Stack unwind information not available. Following frames may be wrong.
01fbff78 781329bb AcroRdIF+0x1c2a
01fbffb0 78132a47 msvcr80!_callthreadstartex+0x1b
01fbffb8 77e6482f msvcr80!_threadstartex+0x66
01fbffec 00000000 kernel32!BaseThreadStart+0x34
 
Only thread 6 was using CPU:
0:000> !runaway
 User Mode Time
  Thread       Time
   6:2d0       0 days 0:00:09.156
   3:748       0 days 0:00:00.203
   5:eb8       0 days 0:00:00.000
   4:ebc       0 days 0:00:00.000
   2:320       0 days 0:00:00.000
   1:240       0 days 0:00:00.000
   0:ea4       0 days 0:00:00.000
0:000> lmv mAcroRdIF
start    end        module name
10000000 1001c000   AcroRdIF   (export symbols)       AcroRdIF.dll
    Loaded symbol image file: AcroRdIF.dll
    Image path: C:\Program Files\Adobe\Reader 9.0\Reader\AcroRdIF.dll
    Image name: AcroRdIF.dll
    Timestamp:        Thu Jun 12 14:12:39 2008 (4850BE57)
    CheckSum:         00023687
    ImageSize:        0001C000
    File version:     9.0.0.0
    Product version:  9.0.0.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Adobe Systems, Inc.
    ProductName:      Adobe PDF IFilter
    InternalName:     PDFFilter
    OriginalFilename: AcroRdIF.dll
    ProductVersion:   9.0.0.0
    FileVersion:      9.0.0.0
    FileDescription:  PDF IFilter
    LegalCopyright:   Copyright 2000-2006 Adobe Systems, Inc.