Software for X-ray data analysis

h1. Software for X-ray data analysis

{{>toc}}

This is an (incomplete) list of necessary and useful software to reprocess and analysis modern X-ray data. If you like to know in more detail on how to process X-ray data, have a look here: Processing X-ray data

Most packages are tested with Linux (OpenSUSE Leap 42.3) but in principle should also work with MacOS. (If not, make a remark in this wiki-page).

h2. Online Tools

* WebPIMMS: https://heasarc.gsfc.nasa.gov/cgi-bin/Tools/w3pimms/w3pimms.pl
	+ Simple tool to convert between physical and instrumental units of X-ray detectors
	+ For Chandra, you want to use this (takes the ageing of the detectors into account): http://cxc.harvard.edu/toolkit/pimms.jsp
	+ ROSAT Instrumental Units: https://heasarc.gsfc.nasa.gov/Tools/xraybg\_help.html#webpimms

* WebSpec: https://heasarc.gsfc.nasa.gov/webspec/webspec.html
	+ Web interface for XSPEC (see below), which simulates spectral data for a variety of X-ray missions/instruments.

* RISA: Remote Interface for Science Analysis (RISA) for XMM-Newton data https://nxsa.esac.esa.int/nxsa-web/#risaHelp

h2. General

* HEASOFT: Collection of many packages. https://heasarc.gsfc.nasa.gov/lheasoft/download.html
	+ You probably just need it for \*XSPEC\* (see below) and \*FTOOLS\* (collection of simple, robust and useful tools to manipulate and analysis Fits-Files).
	+ It also contains optional, mission related software packages, such as for NuSTAR, ROSAT, Suzaku, Swift
	+ Simple rule: initialize the HEASOFT environmental variables in your shell before you initialize other packages (e.g. SAS or CIAO) in order to avoid conflicts
	+ \*Installing HEASOFT:\*

1. 1. 1. How to install on Linux (OpenSUSE Leap 4.23) (as of 11/2017)
		2. {{collapse(Installing from source code for SAS 17.0.0 )

\_(this assumes that SAS is already installed)\_

1. 1) download most recent source code here: http://heasarc.gsfc.nasa.gov/lheasoft/download.html
2. 2) decompress: `cshell > tar -zxvpf heasoft-6.25src2.tar.gz`
3. 3) `cshell > cd heasoft-6.25/BUILD\_DIR`
4. 4) setting PERL environment \*(needs to be at least version 5.24.1)\* : e.g. `cshell > setenv PERL /home/alex/perl5/perlbrew/perls/perl-5.24.1/bin/perl`
5. 5) setting PYTHON environment \*(needs to point to the SAS python version)\* : e.g. `cshell > setenv PYTHON /home/alex/Software/SAS/xmmsas\_20180620\_1732/miniconda3/envs/sas\_17.0.0/bin/python3.6m`
6. 6) `setenv LD\_LIBRARY\_PATH /usr/lib:/usr/local/lib` (not sure if needed for MacOS)
7. 7) Configure `cshell > ./configure --enable-readline --enable-symbols --prefix=/home/alex/Software/heasoft/heasoft-6.25\_install >& config.out` (the install directory is for this case: `/home/alex/Software/heasoft/heasoft-6.25\_install` )
8. 8) check in `config.out` if python folder was correctly found
9. 9) create makefile: `cshell > make >& build.log`
10. 10) check for errors: `cshell > cat build.log | grep "\\*\\*\\*" > build\_error.txt`
11. 11) install: `cshell > make install >& install.log`
12. 12) check for errors: `cshell > cat install.log | grep "\\*\\*\\*" > install\_error.txt`
13. 13) new soft-link: `cshell > ln -s /home/alex/Software/heasoft/heasoft-6.25\_install /home/alex/Software/heasoft/heasoft\_current` \_(This makes it is easier to change the versions of heasoft later)\_
14. 14) If `cshell > setenv HEADAS ~/Software/heasoft/heasoft\_current/x86\_64-pc-linux-gnu-libc2.26/` then you can load HEASOFT with `cshell > source "${HEADAS}/headas-init.csh"`
15. 15) delete file `Xspec.init` before starting new Xspec: `cshell >rm $HOME/.xspec/Xspec.init`

}} (as of 06.05.2019)

h2. Visualization

* ds9: Astronomical imaging and data visualization application (included in CIAO) http://ds9.si.edu/site/Home.html
	+ Can also create an image out of a Event-List (from XMM-Newton or Chandra)
	+ Can be used in combination with tasks of SAS and CIAO (see SAS and CIAO documentations).
	+ Very useful for defining complex region-files.
	+ Can be used within Python: https://hea-www.harvard.edu/RD/pyds9/
	+ Scripting possible. See http://ds9.si.edu/doc/user/scripts/index.html and http://ds9.si.edu/doc/ref/xpa.html

* ChIPS: Chandra Imaging and Plotting System, for press releases (part of CIAO) http://cxc.harvard.edu/chips/

h2. Instrument related

h3. Chandra (and partially also XMM-Newton)

* CIAO: Chandra Interactive Analysis of Observations http://cxc.harvard.edu/ciao/
	+ \*Advice:\* Many CIAO tasks work with XMM-Newton data and you may want to use it rather than SAS, since CIAO is a much more advanced software package and many task are more powerful and robust, e.g. if you have complex mask. Moreover, the documentation is excellent. Installation is also easier.
* Install CIAO before SAS ( http://cxc.harvard.edu/ciao/download/ )
* CIAO Helpdesk http://cxc.cfa.harvard.edu/helpdesk/
* Advice: initialize the HEASOFT before CIAO but CIAO before SAS in the shell in order to avoid conflicts
* CIAO tasks can be executed in parallel with a small trick. Ask "Alex":mailto:alex@kolodzig.eu for details.

* Instrumental background files for ACIS detector (\_stowed\_)
	+ Explanation: http://cxc.harvard.edu/contrib/maxim/stowed/
	+ Files: http://cxc.cfa.harvard.edu/contrib/maxim/acisbg/data/
	+ Also see: Hickox and Markevitch 2006 http://adsabs.harvard.edu/abs/2006ApJ...645...95H

h3. XMM-Newton

* Download and install SAS https://www.cosmos.esa.int/web/xmm-newton/download-and-install-sas
	+ read  How to install on Linux (OpenSUSE Leap 4.23) for more details (as of 11/2017)
	+ HEASOFT Software (see above) should be installed and setup beforehand in any case
	+ \*Follow the list of software requirements very carefully\* (version numbers are important!): https://www.cosmos.esa.int/web/xmm-newton/sas-requirements
* How to use SAS https://www.cosmos.esa.int/web/xmm-newton/how-to-use-sas
* SAS common threads https://www.cosmos.esa.int/web/xmm-newton/sas-threads
* SAS Helpdesk: http://xmm2.esac.esa.int/xmmhelp?user=guest
* Advice: initialize the HEASOFT and CIAO before SAS in the shell in order to avoid conflicts

* Diffuse emission: \_Cookbook for analysis procedures for xmm-newton observations of extended objects and the diffuse background\_ https://heasarc.gsfc.nasa.gov/docs/xmm/esas/cookbook/xmm-esas.html
	+ Please, talk to "Alex":mailto:alex@kolodzig.eu before following this guide. Some things are outdated.

* Background Analysis: https://www.cosmos.esa.int/web/xmm-newton/background (see also the papers in the reference list)
	+ More recent references:
		- A systematic analysis of the XMM-Newton background: I. Dataset and extraction procedures http://adsabs.harvard.edu/abs/2017ExA....44..297M
		- A systematic analysis of the XMM-Newton background: II. Properties of the in-Field-Of-View excess component http://adsabs.harvard.edu/abs/2017ExA....44..309S
		- A systematic analysis of the XMM-Newton background: III. Impact of the magnetospheric environment http://adsabs.harvard.edu/abs/2017ExA....44..273G
		- Ghirardini et al. 2018 (see Appendix A) http://adsabs.harvard.edu/abs/2018A%26A...614A...7G

* Instrumental background files (\_Filter Wheel Closed data, short FWC\_): https://www.cosmos.esa.int/web/xmm-newton/filter-closed

h2. Energy spectrum analysis

* XSPEC: X-Ray Spectral Fitting Package (part of HEASOFT) https://heasarc.gsfc.nasa.gov/lheasoft/xanadu/xspec/index.html
	+ Scripting:
		- in python with the PyXspec package (part of HEASOFT) https://heasarc.gsfc.nasa.gov/lheasoft/xanadu/xspec/python/html/index.html
		- with TCL ( https://heasarc.nasa.gov/docs/xanadu/xspec/xspec11/manual/node65.html ) but may be a bit more complicated than python. ("Alex":mailto:alex@kolodzig.eu has some scripts for automatic fitting of many observations)
	+ There is facebook group for questions: https://www.facebook.com/groups/320119452570/about/
	+ Note: most complex X-ray models are develop specifically for XSPEC. Hence, it's worth installing XSPEC to have wide range of models. However, those models can be accessed also by other software (see below).
	+ Reference: http://adsabs.harvard.edu/abs/1996ASPC..101...17A
	+ {{collapse( Installing the latest patches for XPSEC )

\_(this assumes that HEASOFT has been installed from the source code)\_

1. 1) `$HEADAS` has to be set
2. 2) load HEASOFT (e.g. `cshell > source ${HEADAS}/headas-init.csh` )
3. 3) go to https://heasarc.gsfc.nasa.gov/docs/xanadu/xspec/issues/issues.html
4. 4) download "patch\_install\_4.10.tcl" into folder `heasoft-6.25/Xspec/src/` (That's the folder of the source code!) Direct link: https://heasarc.gsfc.nasa.gov/docs/xanadu/xspec/issues/patch\_install\_4.10.tcl
5. 5) download "Xspatch\_121001f.tar.gz" into folder `heasoft-6.25/Xspec/src/` (That's the folder of the source code! Do not unpack file!) Direct link: https://heasarc.gsfc.nasa.gov/docs/xanadu/xspec/issues/Xspatch\_121001f.tar.gz
6. 6) install patch: `cshell > tclsh patch\_install\_4.10.tcl >& patch\_install.log` (takes a while)
7. 7) check for errors: `cshell > cat patch\_install.log | grep "\\*\\*\\*" > patch\_install\_error.txt`

}} (as of 06.05.2019)

* ISIS (XSPEC alternative): the Interactive Spectral Interpretation System : http://space.mit.edu/home/mnowak/isis\_vs\_xspec/index.html
	+ See the "lectures":http://space.mit.edu/~mnowak/Astronomy\_Schools.html of Michael Nowak on how to use it

* Sherpa: X-Ray Spectral Fitting Package (part of CIAO): http://cxc.harvard.edu/sherpa/
	+ Scripting in python
	+ XSPEC models can be used in Sherpa

* BAX: Bayesian Parameter Estimation and Model comparison with XSPEC and Sherpa: https://johannesbuchner.github.io/BXA/index.html
	+ Reference: http://adsabs.harvard.edu/abs/2014A%26A...564A.125B

* PINTofALE: Package for Interactive Analysis of Line Emission https://hea-www.harvard.edu/PINTofALE

* GRPPHA: simple tool to bin energy channels (part of HEASOFT) https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/summary/cal\_sw\_93\_010\_summary.html
	+ XSPEC can't do this on its own. Useful for low S/N spectra

* Atomic databases for spectroscopic diagnostics:
	+ AtomDB: is an atomic database designed for X-ray plasma spectral modeling http://www.atomdb.org/
		- XSPEC already includes this database but usually not the most recent version. Hence, you only need this if you want to do state of the art spectral fitting,
	+ CHIANTI: Atomic Database for Spectroscopic Diagnostics of Astrophysical Plasmas http://www.chiantidatabase.org/
	+ SPEX is a software package optimized for the analysis and interpretation of high-resolution cosmic X-ray spectra https://www.sron.nl/astrophysics-spex