Fisheye/Crucible - incremental indexing after pulling each commit from Mercurial repository.
Asked by Tatiana Shambereva:

Fisheye/Crucible - incremental indexing after pulling each commit from Mercurial repository.  
It lasts for two hours.

Why do we need this?  
How can I disable reindexing?  
Why don't we index only the files from the commit?

Fisheye/Crucible Version:4.4.1


Answers:
author: Grzegorz Lewandowski
body: Incremental indexing and full reindex are two different things:

\* incremental indexing is the thing you'd like to have -\> pull newest changes and index new commits only

\* reindex means dropping current repository indexes and indexing repository from scratch

In order to see updates on a repository in Fisheye you need incremental index. In order to determine why it takes 2h first of all determine if Fisheye commences incremental indexing or full reindex. You'll be able to find it in debug logs.  

If it's really reindex then I suppose somebody is triggering that in some way (plugin, rest webhook etc)   

If it's incremental index then I suggest enabling debug logs and determining which part takes 2h. Is it connectivity issue and Fisheye is having hard time to pull changes up ? Is this a big changeset with massive changes therefore indexing takes its time ? Hard to tell at the moment.  

The third option that comes to my mind is the fact that potentially you have repository polling set to every 2 hours in Fisheye. This means Fisheye will look for new changes in repository every 2h, effectively it seems like a lot of time for Fisheye to pick up changes but in fact it's not, it's jus the polling period.

comments: author: Tatiana Shambereva
body: Thank you, Grzegorz.

Mercurial - Non-CVS Repository.

**Polling Interval - 60s**

### **User-installed add-ons** {#toc-hId-1566402528}

**No user-installed add-ons found in your FishEye instance.**

**logs**:

2017-09-22 18:33:56,572 DEBUG \[153:StdOutHandler DPC \] fisheye RepositoryStatus-setMessage - Status change \[DPC\]: pulling from http://....

...

2017-09-22 18:33:56,572 DEBUG \[153:StdOutHandler DPC \] fisheye RepositoryStatus-setMessage - Status change \[DPC\]: added 1 changesets with 1 changes to 1 files  
2017-09-22 18:33:56,772 DEBUG \[153:StdOutHandler DPC \] fisheye RepositoryStatus-setMessage - Status change \[DPC\]: (run 'hg update' to get a working copy)  
2017-09-22 18:33:56,888 DEBUG \[IncrPing3 DPC \] fisheye DvcsProcessMonitor-onAfterFinished - Finished process: /usr/bin/hg --config ui.verbose=false pull http://..... took 569ms  
2017-09-22 18:33:56,889 DEBUG \[IncrPing3 DPC \] fisheye DvcsProcessMonitor-onBeforeStart - Starting process: /usr/bin/hg --config ui.verbose=false branches -c  
2017-09-22 18:33:57,123 DEBUG \[IncrPing3 DPC \] fisheye DvcsProcessMonitor-onAfterFinished - Finished process: /usr/bin/hg --config ui.verbose=false branches -c took 234ms  
2017-09-22 18:33:57,124 DEBUG \[IncrPing3 DPC \] fisheye RepositoryStatus-setMessage - Status change \[DPC\]: Updated from http://....  
2017-09-22 18:33:57,124 INFO \[IncrPing3 DPC \] fisheye BaseRepositoryScanner-ping - processing repository DPC (DPC)  
2017-09-22 18:33:57,124 INFO \[IncrPing3 DPC \] fisheye LifecycleRepositoryScanner-doSlurpTransaction - Starting slurp of DPC (DPC)  
2017-09-22 18:33:57,124 DEBUG \[IncrPing3 DPC \] fisheye RepositoryStatus-setMessage - Status change \[DPC\]: Indexing has commenced for repository DPC  
2017-09-22 18:33:57,125 DEBUG \[IncrPing3 DPC \] fisheye Timer-output - -\> slurp of DPC mem_usage = 401219968 heapsize = 608698368 freemem = 207478400

...

...

**and then a lot of lines...**

2017-09-22 18:35:07,413 DEBUG \[IncrPing3 DPC \] fisheye DvcsProcessMonitor-onAfterFinished - Finished process: /usr/bin/hg --config ui.verbose=false annotate -r 1b3fc7713b094f6fbe4df8d484752768ef7439e1 -c -f -l ....cs took 151ms  
2017-09-22 18:35:07,439 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-3:22760, 3-4:22943, 4-11:22760, 11-14:22913, 14-15:22760, 15-18:32712, 18-23:22760, 23-24:22913, 24-28:22760, 28-29:22913, 29-30:22760, 30-31:25436, 31-33:22760, 33-34:22913, 34-35:22760, 35-36:25436, 36-37:22913, 37-38:25436, 38-40:22760, 40-41:22943, 41-42:22760, 42-46:22943, 46-47:25436, 47-70:22943, 70-71:22760, 71-75:22943, 75-76:22760, 76-78:22943, 78-79:22913, 79-81:22943, 81-82:22913, 82-83:22760, 83-91:22943, 91-103:22760, 103-104:23030, 104-116:23086, 116-117:23030, 117-118:32712, 118-121:23030, 121-122:32712, 122-124:23030, 124-126:22760\], hunks=\[\<5:0, 5:1\>, \<30:1, 31:2\>, \<35:1, 37:5\>, \<37:1, 43:1\>\], hunksRevId=71856  
2017-09-22 18:35:07,451 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-1:2072, 1-2:25438, 2-3:2072, 3-4:2088, 4-5:2057, 5-8:22224, 8-9:32717, 9-10:28228, 10-14:25438, 14-15:24257, 15-16:53075, 16-17:22224, 17-18:22765, 18-20:24257, 20-21:22765, 21-22:22947, 22-23:23653, 23-25:28228, 25-26:22224, 26-27:2057\], hunks=\[\<11:1, 11:0\>, \<15:1, 14:0\>, \<17:1, 15:0\>\], hunksRevId=71857  
2017-09-22 18:35:07,455 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-39:54472, 39-40:60038, 40-46:54472, 46-47:57607, 47-58:54472, 58-59:60038, 59-100:54472\], hunks=\[\<96:1, 96:1\>\], hunksRevId=71858  
2017-09-22 18:35:07,459 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-2:53538, 2-3:54490, 3-4:53538, 4-5:54490, 5-7:53538, 7-8:54490, 8-10:53538, 10-11:57053, 11-13:53538, 13-14:60046, 14-16:53538, 16-17:60148, 17-18:60248, 18-19:66655, 19-22:53538, 22-23:54490, 23-24:53538, 24-25:60046, 25-31:53538, 31-32:58192, 32-34:53538, 34-35:56582, 35-36:53538, 36-39:65913, 39-43:53538, 43-76:54490, 76-78:53538, 78-79:54490, 79-92:53538, 92-93:60046, 93-95:53538, 95-96:65970, 96-97:54336, 97-99:59471, 99-100:54336, 100-102:59471, 102-103:54336, 103-104:59471, 104-105:65970, 105-107:54336, 107-108:65970, 108-109:54336, 109-111:59471, 111-112:54336, 112-113:59471, 113-114:65970, 114-115:54336, 115-147:54464, 147-155:57053, 155-156:55516, 156-160:54464, 160-161:60242, 161-162:55516, 162-164:54464, 164-165:58192, 165-169:54464, 169-172:66655, 172-176:58192, 176-177:66655, 177-187:58192, 187-188:56582, 188-194:54464, 194-195:59477, 195-196:54464, 196-197:59477, 197-198:54464, 198-215:59477, 215-217:54464, 217-219:59477, 219-220:54464, 220-227:59477, 227-229:54464, 229-230:58192, 230-231:54464, 231-232:58192, 232-233:54464, 233-234:55516, 234-235:58192, 235-236:54464, 236-253:54490, 253-255:53538\], hunks=\[\<10:0, 10:1\>, \<241:0, 242:1\>, \<242:1, 244:1\>\], hunksRevId=71859  
2017-09-22 18:35:07,467 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-2:2098, 2-3:1900, 3-4:2098, 4-5:2111, 5-8:2098, 8-9:52626, 9-10:32832, 10-11:39831, 11-12:37875, 12-14:39831, 14-15:32832, 15-17:39831, 17-18:37875, 18-19:52626, 19-20:39831, 20-36:68903, 36-37:32832, 37-38:39831, 38-39:52626, 39-40:22773, 40-41:39831, 41-42:52626, 42-43:25218, 43-47:52626, 47-48:25218, 48-50:39831, 50-51:22773, 51-53:39831, 53-54:22773, 54-56:39831, 56-58:22773, 58-59:39831, 59-60:52626, 60-61:22773, 61-62:39831, 62-63:52626, 63-64:2485, 64-65:52626, 65-66:54325, 66-67:22773, 67-69:52626, 69-70:39831, 70-71:52626, 71-72:25218, 72-76:52626, 76-77:28241, 77-79:52626, 79-80:2499, 80-81:39831, 81-82:39869, 82-83:52626, 83-84:39017, 84-85:2557, 85-86:2499, 86-87:39831, 87-88:54325, 88-100:53643, 100-101:58313, 101-102:53080, 102-103:58201, 103-104:39831, 104-108:58201, 108-109:53080, 109-111:58201, 111-112:53080, 112-115:58201, 115-116:54325, 116-120:52626, 120-121:39831, 121-123:52626, 123-125:25218, 125-126:52626, 126-127:58313, 127-137:52626, 137-138:39869, 138-139:39831, 139-146:52626, 146-147:39869, 147-148:39831, 148-149:52626, 149-151:39831, 151-153:52626, 153-154:39831, 154-158:52626, 158-159:39869, 159-160:39831, 160-167:52626, 167-168:39869, 168-169:39831, 169-170:52626, 170-172:39831, 172-174:52626, 174-175:39831, 175-179:52626, 179-180:39869, 180-181:39831, 181-188:52626, 188-189:39869, 189-190:39831, 190-191:52626, 191-193:39831, 193-195:52626, 195-196:39831, 196-200:52626, 200-201:39869, 201-202:39831, 202-209:52626, 209-210:22231, 210-211:39831, 211-218:52626, 218-219:22773, 219-220:39831, 220-223:52626, 223-224:39831, 224-227:52626, 227-228:28241, 228-229:39831, 229-230:52626, 230-232:39831, 232-233:52626, 233-235:39831, 235-236:52626, 236-237:22773, 237-238:39869, 238-241:52626, 241-242:39831, 242-244:52626, 244-245:39831, 245-248:52626, 248-249:22773, 249-250:39869, 250-251:52626, 251-252:39869, 252-254:52626, 254-255:39831, 255-257:52626, 257-258:39831, 258-267:52626, 267-268:39869, 268-269:39831, 269-276:52626, 276-277:23661, 277-278:39869, 278-281:52626, 281-282:39831, 282-285:52626, 285-286:30122, 286-287:39869, 287-288:52626, 288-289:39831, 289-290:39869, 290-291:52626, 291-292:39831, 292-293:39869, 293-294:52626, 294-295:23661, 295-296:39869, 296-297:52626, 297-298:39831, 298-300:52626, 300-301:39831, 301-303:52626, 303-304:39831, 304-313:52626, 313-314:39869, 314-315:39831, 315-322:52626, 322-324:39869, 324-325:52626, 325-326:39831, 326-327:39869, 327-328:52626, 328-329:39831, 329-330:39869, 330-333:52626, 333-334:58313, 334-340:52626, 340-341:28821, 341-347:58297, 347-348:58313, 348-349:39831, 349-350:52626, 350-351:22773, 351-353:52626, 353-354:39831, 354-355:52626, 355-356:53634, 356-357:39831, 357-358:53643, 358-359:53634, 359-360:39831, 360-361:52626, 361-362:53634, 362-363:22773, 363-364:52626, 364-365:25218, 365-370:52626, 370-372:25218, 372-373:52626, 373-374:22773, 374-376:52626, 376-377:22773, 377-378:52626, 378-379:25447, 379-384:52626, 384-386:25447, 386-387:52626, 387-388:25447, 388-393:52626, 393-394:25447, 394-395:22773, 395-396:52626, 396-397:22950, 397-398:39831, 398-399:52626, 399-400:22987, 400-401:39831, 401-402:52626, 402-403:22987, 403-404:39831, 404-405:52626, 405-406:22987, 406-407:39831, 407-408:52626, 408-409:22987, 409-410:22950, 410-411:52626, 411-412:22987, 412-413:39831, 413-414:52626, 414-415:22987, 415-416:39831, 416-417:52626, 417-418:30116, 418-419:23052, 419-420:52626, 420-421:22773, 421-422:22950, 422-423:52626, 423-424:22927, 424-425:39831, 425-426:39869, 426-427:22927, 427-428:39831, 428-429:39869, 429-430:22927, 430-431:39831, 431-432:39869, 432-434:22927, 434-435:22950, 435-436:52626, 436-437:23661, 437-438:39869, 438-439:22950, 439-440:39831, 440-441:39869, 441-442:24264, 442-443:39831, 443-444:39869, 444-445:23052, 445-446:23661, 446-447:52626, 447-448:22773, 448-449:52626, 449-450:22773, 450-463:66269, 463-465:2098, 465-467:1900, 467-468:2098, 468-475:24138, 475-478:1900, 478-479:24138, 479-480:64464, 480-481:28241, 481-482:1900, 482-483:2098, 483-484:1900\], hunks=\[\<142:1, 142:0\>\], hunksRevId=71860  
2017-09-22 18:35:07,473 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-13:44880, 13-14:62369, 14-18:44880, 18-19:45186, 19-20:62704, 20-23:45186, 23-29:44880, 29-43:45186, 43-44:62369, 44-48:44880, 48-49:45186, 49-50:62704, 50-53:45186, 53-60:44880, 60-73:45186, 73-79:45692, 79-80:62704, 80-81:45692, 81-82:45229, 82-93:45692, 93-94:45229, 94-96:44880\], hunks=\[\<29:2, 29:0\>\], hunksRevId=71861  
2017-09-22 18:35:07,480 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-3:29250, 3-4:33159, 4-5:29250, 5-6:71677, 6-9:44459, 9-10:29250, 10-11:29851, 11-12:29250, 12-13:71677, 13-14:31838, 14-19:29250, 19-20:71677, 20-23:29851, 23-24:62705, 24-26:29851, 26-27:62705, 27-41:29851, 41-46:31281, 46-48:29851, 48-49:71677, 49-61:29851, 61-62:62705, 62-68:29851, 68-69:31838, 69-70:60461, 70-71:29250, 71-73:31838, 73-74:34505, 74-76:31838, 76-80:34505, 80-83:43666, 83-84:69440, 84-86:43666, 86-87:69440, 87-88:43666, 88-89:65517, 89-90:43666, 90-91:62705, 91-94:69440, 94-96:43666, 96-97:34505, 97-98:71677, 98-104:34505, 104-107:31838, 107-108:29851, 108-109:71677, 109-110:29250, 110-111:29421, 111-112:71697, 112-114:31838, 114-115:71677, 115-119:71697, 119-120:71677, 120-121:31281, 121-123:29421, 123-124:71697, 124-125:31838, 125-126:29421, 126-127:71677, 127-131:71697, 131-132:71677, 132-135:31838, 135-136:71677, 136-138:31838, 138-139:71677, 139-141:71634, 141-142:71677, 142-146:71634, 146-148:31838, 148-151:71634, 151-152:71677, 152-153:71634, 153-155:31838, 155-157:71634, 157-161:31838, 161-172:71677, 172-173:71697, 173-175:31838, 175-176:71677, 176-178:31838, 178-182:71697, 182-184:31838, 184-186:71634, 186-188:31838, 188-190:29250, 190-193:34090, 193-194:71677, 194-196:34090, 196-197:62705, 197-199:44249, 199-201:34090, 201-204:60461, 204-209:71677, 209-214:60461, 214-224:37483, 224-232:37585, 232-233:44459, 233-234:71677, 234-235:44459, 235-236:71677, 236-240:44459, 240-243:71677, 243-250:44459, 250-255:71677, 255-261:44459, 261-262:71677, 262-265:44459, 265-266:62705, 266-271:44459, 271-272:71677, 272-280:44459, 280-281:62705, 281-295:44459, 295-296:62705, 296-310:44459, 310-311:62705, 311-316:44459, 316-317:71677, 317-325:44459, 325-326:62705, 326-340:44459, 340-341:62705, 341-346:44459, 346-347:71677, 347-348:44459, 348-349:29250, 349-350:71677\], hunks=\[\<6:0, 6:1\>, \<172:0, 173:11\>\], hunksRevId=71862  
2017-09-22 18:35:07,533 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-3:21752, 3-4:43049, 4-5:21866, 5-6:21752, 6-7:33160, 7-8:21752, 8-9:38163, 9-10:43049, 10-13:21752, 13-15:70491, 15-16:43049, 16-17:50824, 17-18:43049, 18-21:21752, 21-22:71609, 22-24:21752, 24-27:51734, 27-35:21752, 35-36:21866, 36-39:70491, 39-40:43049, 40-45:70491, 45-65:43049, 65-68:43563, 68-82:43049, 82-84:43563, 84-85:43049, 85-101:43563, 101-106:43049, 106-111:34506, 111-120:23600, 120-121:32671, 121-122:28282, 122-128:23600, 128-129:23550, 129-132:28427, 132-135:55024, 135-136:28427, 136-141:55024, 141-142:28427, 142-143:50824, 143-144:29852, 144-146:70491, 146-148:28427, 148-160:50824, 160-162:51734, 162-166:50824, 166-180:51734, 180-183:50824, 183-192:32671, 192-194:65754, 194-208:32671, 208-210:65754, 210-216:32671, 216-217:38163, 217-222:32671, 222-224:38163, 224-235:32671, 235-237:65754, 237-247:32671, 247-249:65754, 249-265:32671, 265-267:65754, 267-273:32671, 273-275:65754, 275-286:32671, 286-313:50824, 313-318:70960, 318-320:50824, 320-325:70960, 325-333:50824, 333-360:70491, 360-362:21752\], hunks=\[\<8:0, 8:1\>\], hunksRevId=71863  
2017-09-22 18:35:07,535 DEBUG \[IncrPing3 DPC \] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=\[0-2:71683, 2-3:71610, 3-5:71683, 5-12:71610, 12-14:71683, 14-16:71610\], hunks=\[\<5:0, 5:1\>, \<14:0, 15:1\>\], hunksRevId=71864


author: Tatiana Shambereva
body: Grzegorz, how to fix it?


author: Grzegorz Lewandowski
body: Sorry, I'm not sure what's there to fix.

Name of the thread *IncrPing3 DPC* indicates it's incremental indexing, so I think Fisheye indexes only new changes.

I don't know how much repository slurp takes on your instance. There should be a INFO logs indicating how much time it's take for Fisheye to index a single set of changes (it may be multiple commits at once)

*fisheye BaseRepositoryScanner-ping - done slurp for repository repository (repository) time = 8732 ms*

The logs you've linked (*DefaultBlameChunksCalculator*) are related to a [blame calculations](https://confluence.atlassian.com/fisheye/blame-calculations-785618348.html). You can turn them off if you really want to but it will impact some of the Fisheyes features.