Crucible "on-line" backup?
Asked by Sergey Osokin:

Hello,

according to official documentation there is no way to use Fisheye/Crucicble until the backup completes.

The question is: is it possible to implement "online backup" for the application, which doesn't impact users/UI for backup time?

Thanks.


Answers:
author: Tini Good
body: Do you really need 100% uptime? Our backup via built-in FE/Cruc tool is configured to include everything but repo/app caches (since those can be recreated) and runs at 1:00 a.m. nightly. Records show creating that backup zip only takes 4-5 minutes to complete execution.

What to do with the zip once you've created it is fun. This Windows batch script has error handling, is configured for FE/Cruc **and** JIRA **and** the SQL database, **and** performs service stop/restarts to ensure MySQL database copy is good, **and** checks remaining server freespace, **and** then moves backup zips to a network folder on another server so it's not wasting server space or un-retrievable in the event of a server crash. It **also** does a check on the size of the backup so can confirm easily that backup was good. It auto-deletes backup files for FE/Cruc after 3 days and JIRA after 5 days, since we should have a copy on tape by then via enterprise server backup tools. It keeps a log of the files that were deleted just in case we ever have to pull anything from tape (need filename).

Windows Task Scheduler runs it at 2 am and then auto-emails the log to me on completion, and I've never seen it take longer than 10 minutes total. The thing that takes the longest is probably the free space check since it isn't very efficient. Not the cleanest script of all time, but it will output any errors to the log file, plus the output log is descriptive and formatted for readability. It uses a free Sysinternals tool I placed in the root of C (c:\\du.exe) to accomplish the file size checks.

So, all that, and time offline is \~15 minutes total per night. Our I/O speeds on the server are poor (although we have 10gb between servers) so it may be even faster for you.

```
echo on setlocal enableextensions enabledelayedexpansionset?destdir=\\domainname.com\project\SVN_BK\Atlassian\Backup%random% md %destdir% md %destdir%\jira_dataset logfile=%destdir%\log-atlassian.txt set mailcopy=c:\log-atlassian.txt set logdelete=c:\deletionlog-atlassian.txt?rem This gets the current date and formats it as YYYY_MM_DD to match Atlassian's format set atlasdate=%date:~10,4%_%date:~4,2%_%date:~7,2%?set jiraFolder="C:\Program Files\Atlassian\Application Data\JIRA\data" set jiraBackupFolder=%destdir%\jira_dataset cruc?Service="Atlassian Crucible" set jiraService=JIRA161214003955?set FECrucbackup=D:\FISHEYE_INST\backup\fisheyeandcrucible_backup_%atlasdate%.zip?rem Splitting the folder in two variables allows for handling folders with spaces cleanly with mysqldump set mySQLFolder="C:\Program Files\MySQL\MySQL Server 5.5" set mySQLBackupFolder="C:\Program Files\MySQL\Backups" set result=%mySQLBackupFolder%\jiradbback%atlasdate%.sqlset user=DatabaseAccountWithBackupRightsOnly?echo 1) START Atlassian Backup: %date% %time% &gt;&gt; %logfile% echo . &gt;&gt; %logfile%echo 2a) shutdown Crucible &gt;&gt; %logfile% echo . &gt;&gt; %logfile%?rem As "sc stop" and "net stop" can both return before the service is really stopped, actually rem stopping JIRA can take up to 10 minutes, and we need them stopped before backing up SQL, rem these subroutines repeat the stop request until the services show as STOPPED. rem If they still aren't stopped after 400 attempts, something is wrong with the server, rem and the backup should exit.?set /a counterA=1 :sub_StopCruc for /F "tokens=3 delims=: " %%H in ('sc query %crucService% ^| findstr "        STATE"') do (if /I "%%H" NEQ "STOPPED" (      echo Stopping Atlassian Crucible.      net stop %crucService%      echo Pass %counterA%      set /a counterA+=1      if not counterA == 400 GOTO sub_StopCruc      if counterA == 400 GOTO sub_Error   ) ) 1&gt;&gt;%logfile% 2&gt;&amp;1?echo . &gt;&gt; %logfile% echo 2b) shutdown JIRA &gt;&gt; %logfile%set /a counterB=1 :sub_StopJira for /F "tokens=3 delims=: " %%H in ('sc query %jiraService% ^| findstr "        STATE"') do (if /I "%%H" NEQ "STOPPED" (    echo Stopping Atlassian JIRA.    net stop %jiraService%    echo Pass: %counterB%    set /a counterB+=1    if not counterB == 400 GOTO :sub_StopJira    if counterB == 400 GOTO sub_Error   ) ) 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile%?echo report Atlassian Crucible service state: &gt;&gt; %logfile% sc query %crucService% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo report Atlassian JIRA service state: &gt;&gt; %logfile% sc query %jiraService% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo Crucible and JIRA services should now be stopped, continuing backup... &gt;&gt; %logfile% echo . &gt;&gt; %logfile%?echo 3) creating a backup of jiradb SQL database &gt;&gt; %logfile% "C:\Program Files\MySQL\MySQL Server 5.5\bin\mysqldump.exe" -u %user% -YourPasswordHere jiradb --result-file=%result% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile% echo 3a) checking size of SQL backup &gt;&gt; %logfile% FOR %%I IN (%result%) DO echo %%~zI bytes 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile% echo 4) copying jiradbback.sql to network &gt;&gt; %logfile% xcopy %result% %destdir% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile% echo 4a) delete local copy of %result% since it has been copied to network &gt;&gt; %logfile% del %result% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile%?echo 5) copying JIRA Data folder to network &gt;&gt; %logfile% xcopy %jiraFolder% %jiraBackupFolder% /S 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile%?echo 6) starting JIRA service &gt;&gt; %logfile% net start %jiraService% 1&gt;&gt;%logfile% 2&gt;&amp;1 ?echo 7) starting Crucible service &gt;&gt; %logfile% net start %crucService% 1&gt;&gt;%logfile% 2&gt;&amp;1?rem FE/Cruc backup move echo 8) copying FECruc backup file to network &gt;&gt; %logfile% copy %FECrucbackup% %destdir% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile% ?echo 9) delete local copy of %FECrucbackup% since it has been copied to network &gt;&gt; %logfile% del %FECrucbackup% 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile%?echo 10) END Atlassian Backup Complete: %date% %time% &gt;&gt; %logfile% echo . &gt;&gt; %logfile%?echo 11) Check remaining FreeSpace on Server D Drive: &gt;&gt; %logfile%rem Checking deviceID in case in future this script ever needs to check space for other drives for /f "skip=1 tokens=1-3" %%a in ('wmic logicaldisk get deviceid^,FreeSpace^,Size') do (    set disk=%%a    if "!disk!"=="D:" (       set "FreeBytesDrive=%%b"       set "TotBytesDrive=%%c"       echo(       if defined TotBytesDrive (          set /a Fmb=!FreeBytesDrive:~0,-6!, Tmb=!TotBytesDrive:~0,-6!, Umb=Tmb-Fmb          call :ConvertGb Tgb Tmb          call :ConvertGb Fgb Fmb          call :ConvertGb Ugb Umb?         echo %%a Used:!UGb! - Free:!FGb! / Tot:!TGb! Gb &gt;&gt; %logfile%       )    ) ) echo . &gt;&gt; %logfile%?echo 12) Total final size of backup to %destdir% : &gt;&gt; %logfile% c:\du.exe -q %destdir% /accepteula 1&gt;&gt;%logfile% 2&gt;&amp;1?echo 13) Deleting files and folders more than 3 days old &gt;&gt; %logfile% PushD \\YourDomain.com\project\SVN_BK\Atlassian\ &amp;&amp; (forfiles /D -3 /C "CMD /C if @ISDIR==TRUE echo Backup Directory @relpath created on @fdate deleted on %date% &gt;&gt; %logfile%") &amp; PopD 1&gt;&gt;%logfile% 2&gt;&amp;1 PushD \\YourDomain.com\project\SVN_BK\Atlassian\ &amp;&amp; (forfiles /D -3 /C "CMD /C if @ISDIR==TRUE echo Backup Directory @relpath created on @fdate deleted on %date% &gt;&gt; %logdelete%") &amp; PopD 1&gt;&gt;%logfile% 2&gt;&amp;1 PushD \\YourDomain.com\project\SVN_BK\Atlassian\ &amp;&amp; (forfiles /D -3 /C "CMD /C if @ISDIR==TRUE echo RD /Q @FILE &amp;RD /Q /S @FILE") &amp; PopD 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile%?rem not worried about keeping a log of these automatic JIRA backups echo 14) Deleting daily automatic backups more than 5 days old &gt;&gt; %logfile% PushD C:\Program Files\Atlassian\Application Data\JIRA\export &amp;&amp; (forfiles /D -5 /C "CMD /C if @ISDIR==TRUE echo RD /Q @FILE &amp;RD /Q /S @FILE") &amp; PopD 1&gt;&gt;%logfile% 2&gt;&amp;1 echo . &gt;&gt; %logfile%?rem Copying log to local server so that it is accessible to server's Scheduled Task for email at completion echo 15) Copy logfile to local server for confirmation mail &gt;&gt; %logfile% echo F | xcopy %logfile% %mailcopy% /y /c 1&gt;&gt;%logfile% 2&gt;&amp;1?GOTO :EOF?rem GB with 3 decimal, use only 2 for now, high precision, add padding :ConvertGB   set /a tmp=%2-%2/10+%2*1000/31926   set "tmp=      !tmp:~0,-3!.!tmp:~-3,2!"   set tmp=!tmp:~-6!   set %1=!tmp: .=0.!   exit /b?:sub_Error   if %counterA% == 400 echo "Crucible would not stop, exiting backup script."   if %counterB% == 400 echo "JIRA would not stop, exiting backup script."   exit /b?exit
```

comments: author: Tini Good
body: Oh, and it grabs all of your files JIRA attachments and avatars too.


author: Sergey Osokin
body: Hi Tini,thanks for sharing your code.Yes, we really need 100% uptime for FeCru.Could you provide additional information about instance, where backup basically runs just 5 minutes:1) how many projects/repos you have?2) how long it takes to restore instance from backup?3) same as 2) but with indexes rebuild?Thanks in advance.


author: Tini Good
body: My way definitely takes things offline for a short period. Only reason I can think of to have a 100% uptime requirement is distributed dev team across internal timezones? Unsure if FE/Cruc has support for server replication.1) 20 repos, 18 projects, 50 users2) I do this too infrequently to give you an accurate answer. Primary method of migrating data to a test server before a major version update is via the easier FISHEYE_INST copy (with MySQL dump/restore), rather than actual restore from FE/Cruc zip backup file. 3) \~Once a year I probably force an optimized index rebuild after restart due to performance issues, and that can take 20-40 minutes. But, activity is unrelated to any backup and, our bad virtual server I/O throughput probably doesn't help. Virtual Server host, dual Xeons @ 2.2Ghz (8 cores), 8GB RAM, 64-bit OS. FYI our Fisheye indexes a 122GB Subversion instance, which is also co-hosted.


author: gustavo_refosco
body: Hi Sergey,

Though you could implement your own backup strategy to backup both the FISHEYE_INST directory (which is Crucible's data directory) and your database, hot backups for Crucible are not recommended, as this can lead to inconsistent backups.

The backup tool offered by Crucible stops users from doing anything to avoid new information to be written while the backup is occurring with the purpose of avoiding any inconsistency.

Regards,

Gustavo Refosco

comments: author: Sergey Osokin
body: Hi Gustavo,thank for your quick reply.What I've understood from your comment is that Atlassian can't recommend hot backups.But my question was a bit different - we're looking for "online backup" solution.Is it possible?Thanks.


author: gustavo_refosco
body: Hi Sergey,I'm sorry, I got the concept of "online backup" as a hot backup. Could you please define further what you mean by "online backup"? If this is about being able to backup Crucible without stopping it, this is correspondent to the first method presented in <https://confluence.atlassian.com/display/CRUCIBLE/Backing+up+and+restoring+Crucible+data> through which you can perform the backup via UI - either way you already mentioned the documentation before, so I think you already checked at this.Regards,Gustavo


author: Sergey Osokin
body: Hi Gustavo,thanks for link to documentation, this is what I've found on Atlassian site as recommended solution.When we talking about "online backup" solution, it means no impact for customers, no interrupt for service. Let's say we need 24x7 "Code Review" solution.Any idea?Regards,SergeyI can't send my comment yesterday because of this message:Your activity is currently limited because you've commented, answered or asked a question 3 times in the past 24 hours. These limits apply until you earn 25 points, then you can ask, answer and comment as much as you like.Any chance to add 25 points to my humble person?


author: Sergey Osokin
body: Hello?


Tags: fisheye-crucible-administration