snovault 1.0.68
Snovault Hybrid Object Relational Database Framework
License: MIT

SnoVault JSON-LD Database Framework
===================================

System Installation (OSX Big Sur(testing), Catlina(10.15.x), Mojave(10.14.6))
------------------------------------------------------------------------------

| We will try to keep this updated as OSX, Xcode, and brew update.  However the steps below are
  examples and not guaranteed to work for your specific system.  See the dependency's website for
  detailed instructions or let us know of any changes with a pull request.

1. Command line tools
    .. code-block:: bash

        xcode-select --install

2. brew: https://brew.sh/ . Make sure git is installed

3. Python 3.8.5

4. Postgres\@11 (Postgres\@9.3 also works)
    .. code-block:: bash

        brew install postgresql@11
        # May need to add postgres to PATH in your shell profile, e.g. ~/.bash_profile, ~/.zshrc
        # echo 'export PATH="/usr/local/opt/postgresql@11/bin:$PATH"' >> YOUR_SHELL_PROFILE

5. Node 10.x.x
    .. code-block:: bash

        brew install node@12

    You may need to link ``node``/``npm`` with ``brew link node@12`` then add it to your ``PATH``

6. Ruby - Non system version to install compass while avoiding permission errors
    .. code-block:: bash

        brew install ruby
        # May need to add ruby to your bash_profile/zshrc and restart terminal

7. Compass
    .. code-block:: bash

        gem install compass
        # Test the install
        compass -v
        # If the command is not found then find your ruby bin directory
        ls /usr/local/lib/ruby/gem/
        # If you have two versions use the active one
        ruby -v
        # Using the correct ruby version bin diretory, make a sym link
        ln -s /usr/local/lib/ruby/gems/2.6.0/bin/compass /usr/local/opt/ruby/bin/compass

8. Java 11
    .. code-block:: bash

        brew install openjdk@11
        # Add to your PATH in terminal profile, i.e. ~/.bash_profile or ~/.zshrc
        export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

9. Elasticsearch 5.x
    .. code-block:: bash

        # Download tar: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.0.tar.gz

        # Decompress
        tar -xvf ~/Downloads/elasticsearch-5.6.0.tar.gz -C /usr/local/opt/

        # Add to PATH in your terminal profile, i.e. ~/.bash_profile or ~/.zshrc
        export PATH="/usr/local/opt/elasticsearch-5.6.0/bin:$PATH"

        # Restart terminal and check versions
        elasticsearch -V

10. Brew dependencies
        .. code-block:: bash

            brew install libmagic nginx graphviz redis

11. Chrome driver for Testing

        `Chromedriver <https://chromedriver.chromium.org/downloads>`_ is needed in your PATH.
        If working in a python virtual environment, then the chromedriver can be added to
        your-venv-dir/bin directory.

        You also need to install Chrome (if not already installed).
        In addition, allow ``chromedriver`` (System Preferences->Security & Privacy) to run to run bdd tests

Application Installation
========================

1. Create a virtual env in your work directory. Here we use python3 venv module.  Use venv, like conda, if you please
    .. code-block:: bash

        cd your-work-dir
        python3 -m venv snovault-venv
        source snovault-venv/bin/activate
        pip install -U pip==21.0.1

2. Clone the repo and install requirements
    .. code-block:: bash

        cd snovault
        pip install -e '.[dev]'

    If psycopg2 fails to compile, you may need to set LDFLAGS to the output of ``pg_config --ldflags`` before pip installation.
        .. code-block:: bash

            LDFLAGS=$(pg_config --ldflags) pip install -e '.[dev]'

    If you have errors at runtime that look like this::

        ImportError: dlopen(/Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so, 2): Symbol not found: _PQencryptPasswordConn
        Referenced from: /Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so
        Expected in: /usr/lib/libpq.5.dylib
        in /Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so

    you may need to add the ``brew``-installed Postgres headers, usually ``-L/usr/local/opt/postgresql@11/lib``, to the ``LDFLAGS`` in addition to the ones given by ``pg_config --ldflags``.

3. Build Application
    .. code-block:: bash

        # Make sure you are in the snovault-venv
        make clean && buildout

4. Run Application
    .. code-block:: bash

        # Make sure you are in the snovault-venv
        dev-servers development.ini --app-name app --clear --init --load
        # In a separate terminal, make sure you are in the snovault-venv
        pserve development.ini

5. Browse to the interface at http://localhost:6543

6. Run Tests
    * no argument runs non bdd tests

    .. code-block:: bash

        # Make sure you are in the snovault-venv
        ./circle-tests.sh bdd
        ./circle-tests.sh npm
        ./circle-tests.sh

Changes
=======

Snovault 1.0.68 (released) encoded v130
-----------------------------------------
1. SNO2-25-multiple-types (#402)

Snovault 1.0.67 (released) encoded v129.2
-----------------------------------------
1. SNO2-21-files-crash (#399)

Snovault 1.0.66 (released) encoded v129.0
-----------------------------------------
1. ENCM-89-support-big-image-attachments (#390)

Snovault 1.0.65 (released) encoded v128.0
-----------------------------------------
1. increase-field-limit (#388)

Snovault 1.0.64 (released) encoded v128.0
-----------------------------------------
1. Pin splinter (#385)

Snovault 1.0.63 (released) encoded v128.0
-----------------------------------------
1. ENCM-29-pin-antlr (#379)

Snovault 1.0.62 (released) encoded v124.0
-----------------------------------------
1. SNO-217-register-search-config-pieces-from-item (#370)

Snovault 1.0.61 (released) encoded v124.0
-----------------------------------------
1. SNO-214-upgrade-node-to-version-12-for-circleci-tests (#365)
2. SNO-210-read-columns-from-configs (#368)
3. SNO-216 Fix dev-servers 100% cpu usage when nothing to print. (#367)
4. SNO-212 Allow gzip attachments (#363)
5. SNO-213-circleci-node-install-fix (#364)
6. SNO-211-increase-batch-size-in-scroll-api (#360)

Snovault 1.0.60 (released) encoded v121.0
-----------------------------------------
1. SNO-209-move-search-code-to-independent-package (#356)
2. SNO-206-use-redis-cache-for-rnget-view (#354)
3. SNO-207-facet-groups-support (#355)
4. SNO-205-allow-customization-of-default-sort (#353)

Snovault 1.0.59 (released) encoded v120.0
-----------------------------------------
1. SNO-204-allow-facet-caching (#351)

Snovault 1.0.58 (released) encoded v118.0
-----------------------------------------
1. SNO-170-avoid-rendering-from-database-3 (#346)
2. SNO-202-upgrade-python (#345)

Snovault 1.0.57 (released) encoded v116.0
-----------------------------------------
1. revert SNO-202-python3.8.5-upgrade (#338)
2. SNO-201-add-search-config-registry (#336)
3. SNO-202-python3.8.5-upgrade (#335)

Snovault 1.0.56 (released) encoded v115.0
-----------------------------------------
1. SNO-200-remove-nonsubstring-filter-from-mapping (#332)
2. SNO-199-allow-customization-of-search-fields (#333)
3. SNO-197-add-range-queries-and-min-max-aggs (#329)
4. SNO-198-allow-configuration-of-reserved-search-params (#330)
5. SNO-138-fix-pytest-warnings (#324)
6. SNO-182-readme-update (#323)

Snovault 1.0.55 (released) encoded v114.0
-----------------------------------------
1. SNO-192-update-pip-install (#321)

Snovault 1.0.54 (released) encoded v113.0
-----------------------------------------
1. SNO-27-pip-installable (#316)
2. SNO-191-pyramid-env-vars (#318)
3. SNO-117-search-as-you-type (#220)

Snovault 1.0.53 (released) encoded v112.0
-----------------------------------------
1. SNO-158-create-minimally-indexed-deploy
2. SNO-190-pin-setuptools-in-circle-ci (#315)

Snovault 1.0.52 (released) encoded v111.0
-----------------------------------------
1. SNO-185-Log-connection-error (#311)
2. SNO-183-allow-specification-of-cart-in-search (#310)
3. SNO-181-Update-install-readme (#309)

Snovault 1.0.51 (released) encoded v109.0
-----------------------------------------
1. SNO-174-add-local-storage-redis (#299)
2. SNO-176-invalidate-object-when-unlinked (#303)
3. SNO-177-add-view-with-specified-fields (#300)
4. SNO-178-fix-indexer-test (#304)

Snovault 1.0.50 (released) encoded v108.0
-----------------------------------------
1. SNO-168-stream-heterogeneous-uuids  (#292)
2. SNO-169 Creating a command for admin user creation (#293)

Snovault 1.0.49 (released) encoded v106.0
-----------------------------------------
1. SNO-166-update-query-string (#288)

Snovault 1.0.48 (released) encoded v105.0
-----------------------------------------
1. SNO-163-install-graphviz (#285)

Snovault 1.0.47 (released) encoded v104.0
-----------------------------------------
1. SNO-159-allow-remote-indexing (#282)

Snovault 1.0.46 (released) encoded v103.0 (unreleased)
------------------------------------------------------
1. SNO-160-use-boolean-values-in-facets (#280)
2. SNO-116 Use summary_matrix instead of summary (#278)
3. SNO-157-make-facets-collapsable (#275)

Snovault 1.0.45 (released) encoded v102.0
-----------------------------------------
1. SNO-154-fix-magic-json-bug (#272)
2. SNO-156 Use pyramid_retry 2.1.1 (#273)
3. SNO-137-add-search-doc (#267)
4. SNO-152 fix circle-ci browser install (#269)

Snovault 1.0.44 (released) encoded v99.0
----------------------------------------
1. SNO-150 Remove duplicates and fix antlr version (#266)
2. SNO-149-allow-specification-of-audit-field (#265)
3. SNO-143-update-and-switch-readme
4. SNO-148-uuid-indexing-times-new-line-bit
5. SNO-144-redirect-downloads-through-nginx-or-apache (#192)

Snovault 1.0.43 (released) encoded v98.0
----------------------------------------
1. SNO-146-clean-up-threadmanager-on-pyramid-retry-exception (#261)

Snovault 1.0.42 (released) encoded v97.0
----------------------------------------
1. SNO-145-handle-missing-obj-on-request (#258)

Snovault 1.0.41 (released) encoded v97rc
----------------------------------------
1. SNO-142 Minor fixes for ubutnu 18 on v96x1rc1 (#252)
2. SNO-141-update-osx-catalina-readme (#252)
3. SNO-140 Allow shorting uuids with flag (#249)
4. SNO-139 Log detailed initial indexing times (#249)
5. SNO-135 Update installation for osx catalina (#249)
6. SNO-132 Update to py37 lrowe pr with fixes (#249)

Snovault 1.0.40 (released) encoded v95.0
----------------------------------------
1. SNO-129-add-fielded-generator-response (#244)
2. SNO-128-make-select-calculated-properties-view (#245)
3. SNO-133-setuptools-test-failure (#247)

Snovault 1.0.39 (released) encoded v94.0
----------------------------------------
1. SNO-126-allow-default-value-for-missing-aggregation-field (#242)

Snovault 1.0.38 (released) encoded v93.0
----------------------------------------
1. SNO-125-switch-batch-download-to-query-string (#238)
2. SNO-118-simple-query-string (#240)
3. SNO-123-limit-es-scan (#237)
4. SNO-121-fix-circleci-apt-install (#224)

Snovault 1.0.37 (released) encoded v92.0
----------------------------------------
1. SNO-119-escape-regex-slash (#221)
2. [HOTFIX] SNO-120-stream-large-json (#222)
3. SNO-115-delete-old-search-code (#219)

Snovault 1.0.36 (released) encoded v91.0
----------------------------------------
1. SNO-112-fix-npm-audit-issues (#216)
2. SNO-113-make-audit-view (#217)

Snovault 1.0.35 (UNRELEASED) encoded
----------------------------------------
1. SNO-95-refactor-search (#214)

Snovault 1.0.34 (released) encoded v88.0
----------------------------------------
1. SNO-108-improve-search-new (#212)
2. SNO-105-fix-batchupgrade-tests-sno103 (#211)
3. SNO-106-fix-spec-char-500-error (#209)

Snovault 1.0.33 (released) encoded v87.0
----------------------------------------
1. SNO-103 fix-batch-upgrade-error-logging (#206)
2. SNO-102 use-openjdk-java-for-circleci (#205)

Snovault 1.0.32 (released) encoded v87rc1
-----------------------------------------
1. SNO-99 port travis tests to circleci (#202)
2. SNO-100 Parallelize batchupgrade script (#201)
3. SNO-92 Fix batch upgrade logging (#201)
4. SNO-101 Refactor batchupgrade (#201)

Snovault 1.0.31 (released) encoded v86.0
-----------------------------------------
1. SNO-98 Update sauce connect to 4.5.3 (#199)
2. SNO-93 user-count-does-not-match (#191)
3. SNO-90 added-advancedQuery (#183)

Snovault 1.0.30 (released) encoded v85x2
----------------------------------------
1. SNO-96 fix indexing tests (#196)

Snovault 1.0.29 (released) encoded v85rc1
-----------------------------------------
1. SNO-91 update-travis-java-ref (#188)
2. SNO-87 fixed-issue-while-profile-url-does-not-work (#179)
3. SNO-86 escape-search-text (#178)
4. SNO-88 fix-user-search-count (#180)

Snovault 1.0.28 (released) encoded v84rc1
-----------------------------------------
1. SNO-89 Fix backoff error key in indexer (#181)
2. SNO-85 facet-reappearing (#175)

Snovault 1.0.27 (released) encoded v83.0
----------------------------------------
1. SNO-83 Rotate img attachment for EXIF (#174)

Snovault 1.0.26 (released) encoded v82.0
----------------------------------------
1. SNO-78 add long facet type (#168)
2. SNO-80 typeahead for facets (#168)
3. SNO-73 Add uuid queue module (#162)

Snovault 1.0.25 (released) encoded v81.0
----------------------------------------
1. SNO-77 Add review to submit_for validation (#166)

Snovault 1.0.24 (released) encoded v80.0
----------------------------------------
1. SNO-75-fix-linux-travis-option (#163)

Snovault 1.0.23 (released) encoded v79.0 also
---------------------------------------------
1. SNO-72 Update simple for uuid module (#158)

Snovault 1.0.22 (released) encoded v79.0 also
---------------------------------------------
1. SNO-68 Stop using _all for indexing (#155)
2. SNO-74 Remove npm shrinkwrap json (#157)

Snovault 1.0.21 (released) encoded v78.0 also
---------------------------------------------
1. SNO-65 Refactor indexer uuids as server client (#151)

Snovault 1.0.20 (released) encoded v78.0
----------------------------------------
1. SNO-63 Update pip requests and remove wal-e reqs (#150)
2. SNO-66 Add new endpoint to map schema to schema titles (#152)

Snovault 1.0.19 (released) encoded v77.0
----------------------------------------
1. SNO-60-check-report-res-has-view (#147)
2. SNO-50 Initial shopping cart (#142)
3. SNO-59-fix-index-logger-name (#137)
4. SNO-53 Add index flags to indexers (#137)

Snovault 1.0.18 (released) encoded v76.0
----------------------------------------
1. SNO-49 Change audit inherit default (#132)
2. SNO-31 Refactor search related views (#141) (#143)

1.0.17
1. [HOTFIX] SNO-54-fix-schema-copy-line (#136)

1.0.16
1. SNO-52-alter-select-distinct-values (#131)

1.0.15
1. SNO-48-add-embed-cache-to-ini (#127)

1.0.14
1. SNO-45 Increase embed_capacity (#123)

1.0.13
1. SNO-46 Remove unused search type arg (#122)
2. SNO-43 Clean up snovault startup (#116)

1.0.12
1. SNO-42 Check call count explicitly (#118)

1.0.11
01. SNO-41-put-validator-accession (#112)

1.0.10
01. SNO-35 fix bug in get_rev_links(#111)
02. SNO-40 Upgrade blob storage to boto3 (#110)

1.0.9:
01. SNO-38 Return lists from get_related_uuids (#108, #105)

1.0.8: Released with 1.0.9
01. SNO-36-update-buildout (#104)
02. SNO-34-nginx-dev-proxy-headers (#103)

1.0.7: The only update was reverted.  Empty Release.

1.0.6:
01. SNO-33 specify index for get_by_unique_key from collection (#94)
02. SNO-28 limit ES storage to indices created for snovault resources (#93)

1.0.5:
01. SNO-30 Split Indexer State from indexer file and update
02. SNO-10 Remove audit indexing via 2-pass
03. SNO-9 Add api end points to new indexer meta objects
04. SNO-25 Make uniqueItems to check the serialized values (#85)
05. SNO-26 Add schemas map view to profiles (#86)
06. SNO-29 Limit audits on form update (#87)
07. SNO-19 Update DOI preferred resolver url (#80)

1.0.4:
-SNO-15 Add index to storage propertysheet
-SNO-14 Update delete script

1.0.3:
-SNO-8 Add JSONSchemas type to profiles page (#73)


1.0.2: * Issues discovered while release of ENCD v65 part 2

-SNO-12: Set max clause parameter in es indexer search #75
-SNO-11: Add timeout to ES indexer search query #74

1.0.1: * Issues discovered while release of ENCD v65

-SNO-6: Fix index query too many clauses failure
-SNO-5: Update psycopg to match encoded version 2.7.3

1.0.0:

- 31 delete
- ES5 Fix: Update index settings shard number

0.33:

- ES5 Update: ENCD-2488 ES5 Update aka RM3910
- Fix travis build: Pre Install setuptools with pip for travis like ENCD-3722

0.32:

- Update to ENCD-3669 to not include notSubmittable
  reverse links in the edit view of an object.

0.31:

- ENCD-3684 Specify https index to fix buildout, update
  changelog.

- Use notSubmittable instead of calculatedProperty
  to indicate properties that may not be submitted.
  Related to ENCD-3669.

0.30:

- Document dependency on java 8.

- Disable 2nd indexing pass.

0.29:

- Fix recording indexing errors.

- Add some documentation about indexing.

0.28:

- Add support for adding and updating child objects
  specified as abstract types in the schema.

- Split indexing into 2 phases.

0.27:

- Move embed cache to connection and increase size.

- Fix reporting upgrade errors when error path includes an integer.

0.26:

- Indexer: Limit workers to 1 task and scale chunk size based on number of items being indexed.

0.25:

- Indexer: Limit workers to 4 tasks to avoid out-of-memory errors.

0.24:

- If the schema specifies an explicit `mapping`, use it when building the elasticsearch mapping.  This provides an escape valve for edge cases (such as not indexing the layout structure of a page).

- upgrade to sauceconnect v4.4.4 

- add port_range to wsgi_tests (mrmin)

0.23:
- replace copy.deepcopy() for faster indexing

0.22:
- New version of image magic, fix sauce labs

0.21:
- (pypi errors, identical to 0.22)

0.20:
- Patch mpindexer for better error messages