Manifest

The mobile manifest endpoint provides a cached version of parts of the static information available for Destiny.

## PHP 2.0

This is an updated solution using some of the new knowledge mentioned in the old explanation (see below).

```
<?php
define('COOKIE\_FILE', 'cookie.txt');
define('BUNGIE\_URL', 'https://www.bungie.net');
define('API\_KEY', '{INSERT\_YOUR\_API\_KEY\_HERE}');
define('USER\_AGENT', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1');

define('SETTING\_FILE', 'settings.json');

$default\_options = array(
    CURLOPT\_USERAGENT => USER\_AGENT,
    CURLOPT\_COOKIEJAR => COOKIE\_FILE,
    CURLOPT\_COOKIEFILE => COOKIE\_FILE,
    CURLOPT\_RETURNTRANSFER => true,
    CURLOPT\_FOLLOWLOCATION => true,
    CURLOPT\_SSL\_VERIFYHOST => 2,
);

function loadSettings() {
    if (!file\_exists(SETTING\_FILE)) return new stdClass();
    return json\_decode(file\_get\_contents(SETTING\_FILE));
}
function setSetting($name, $value) {
    $settings = loadSettings();
    $settings->{$name} = $value;
    file\_put\_contents(SETTING\_FILE, json\_encode($settings));
}
function getSetting($name) {
    $settings = loadSettings();
    if (isset($settings->{$name})) return $settings->{$name};
    return '';
}

function parseCookieFile($file) {
    $cookies = array();
    if (file\_exists($\_SERVER['DOCUMENT\_ROOT'].'/'.$file)) {
        $lines = file($file);
        foreach($lines as $line) {
            if (substr\_count($line, "\t") == 6) {
                $tokens = explode("\t", $line);
                $tokens = array\_map('trim', $tokens);

                $domain = preg\_replace('/#[^\_]+\_/i', '', $tokens[0]);
                $flag = $tokens[1] == 'TRUE';
                $path = $tokens[2];
                $secure = $tokens[3] == 'TRUE';
                $expiration = $tokens[4];
                $name = $tokens[5];
                $value = $tokens[6];
                if (!isset($cookies[$domain])) $cookies[$domain] = array();
                $cookies[$domain][$name] = array(
                    'flag' => $flag,
                    'path' => $path,
                    'secure' => $secure,
                    'expiration' => $expiration,
                    'value' => $value
                );
            }
        }
    }
    return $cookies;
}

function doRequest($path) {
    global $default\_options;

    $cookies = parseCookieFile(COOKIE\_FILE);
    $bungieCookies = isset($cookies['www.bungie.net']) ? $cookies['www.bungie.net'] : array();

    $ch = curl\_init(BUNGIE\_URL.$path);
    curl\_setopt\_array($ch, $default\_options);
    curl\_setopt\_array($ch, array(
        CURLOPT\_HTTPHEADER => array(
            'x-api-key: '.API\_KEY,
            'x-csrf: '.(isset($bungieCookies['bungled']) ? $bungieCookies['bungled']['value'] : '')
        )
    ));
    $response = curl\_exec($ch);
    curl\_close($ch);

    return json\_decode($response);
}

function updateManifest($url) {
    $ch = curl\_init(BUNGIE\_URL.$url);
    curl\_setopt\_array($ch, array(
        CURLOPT\_RETURNTRANSFER => true
    ));
    $data = curl\_exec($ch);
    curl\_close($ch);

    $cacheFilePath = 'cache/'.pathinfo($url, PATHINFO\_BASENAME);
    if (!file\_exists(dirname($cacheFilePath))) mkdir(dirname($cacheFilePath), 0777, true);
    file\_put\_contents($cacheFilePath.'.zip', $data);

    $zip = new ZipArchive();
    if ($zip->open($cacheFilePath.'.zip') === TRUE) {
        $zip->extractTo('cache');
        $zip->close();
    }

    $tables = array();
    if ($db = new SQLite3($cacheFilePath)) {
        $result = $db->query("SELECT name FROM sqlite\_master WHERE type='table'");
        while($row = $result->fetchArray()) {
            $table = array();
            $result2 = $db->query("PRAGMA table\_info(".$row['name'].")");
            while($row2 = $result2->fetchArray()) {
                $table[] = $row2[1];
            }
            $tables[$row['name']] = $table;
        }
    }

    return $tables;
}

function checkManifest() {
    // Checking for Manifest changes.
    $result = doRequest('/Platform/Destiny/Manifest/');

    // Grab the path of the language you want
    $database = $result->Response->mobileWorldContentPaths->en;

    // Check to see if had been changed
    if ($database != getSetting('database')) {
        // New database found.
        $tables = updateManifest($database);
        setSetting('database', $database);
        setSetting('tables', $tables);
    }
}

function queryManifest($query) {
    $database = getSetting('database');
    $cacheFilePath = 'cache/'.pathinfo($database, PATHINFO\_BASENAME);

    $results = array();
    if ($db = new SQLite3($cacheFilePath)) {
        $result = $db->query($query);
        while($row = $result->fetchArray()) {
            $key = is\_numeric($row[0]) ? sprintf('%u', $row[0] & 0xFFFFFFFF) : $row[0];
            $results[$key] = json\_decode($row[1]);
        }
    }
    return $results;
}

function getDefinition($tableName) {
    return queryManifest('SELECT \* FROM '.$tableName);
}

function getSingleDefinition($tableName, $id) {
    $tables = getSetting('tables');

    $key = $tables->{$tableName}[0];
    $where = ' WHERE '.(!is\_numeric($id) ? $key.'='.$id.' OR '.$key.'='.($id-4294967296) : $key.'="'.$id.'"');
    $results = queryManifest('SELECT \* FROM '.$tableName.$where);

    return isset($results[$id]) ? $results[$id] : false;
}

checkManifest();

echo '<pre>Get Gjallarhorn: '.json\_encode(getSingleDefinition('DestinyInventoryItemDefinition', 1274330687), JSON\_PRETTY\_PRINT).'</pre>';

echo '<pre>DestinyInventoryBucketDefinition: '.json\_encode(getDefinition('DestinyInventoryBucketDefinition'), JSON\_PRETTY\_PRINT).'</pre>';

```

## PHP

Below is a PHP solution taken from a topic by lowlines that describes how read information from the manifest.

### Response

```
{
 "Response": {
 "version": "45194.15.05.13.1136-2",
 "mobileAssetContentPath": "\/common\/destiny\_content\/sqlite\/asset\/asset\_sql\_content\_7c23eefeaa0003dbc141793135dae141.content",
 "mobileGearAssetDataBases": [
 {
 "version": 0,
 "path": "\/common\/destiny\_content\/sqlite\/asset\/asset\_sql\_content\_7c23eefeaa0003dbc141793135dae141.content"
 },
 {
 "version": 1,
 "path": "\/common\/destiny\_content\/sqlite\/asset\/asset\_sql\_content\_039b4a64b68a7ae75a1d349c301e494c.content"
 }
 ],
 "mobileWorldContentPaths": {
 "en": "\/common\/destiny\_content\/sqlite\/en\/world\_sql\_content\_bafce26a4daec6ba6932e93bb5bd200c.content",
 "fr": "\/common\/destiny\_content\/sqlite\/fr\/world\_sql\_content\_50895b556dfee93a1f3d5a3817c18b19.content",
 "es": "\/common\/destiny\_content\/sqlite\/es\/world\_sql\_content\_d016ba5a8d83b1526212bddaefbd96c6.content",
 "de": "\/common\/destiny\_content\/sqlite\/de\/world\_sql\_content\_52fbe64346a0110511a86238aec07310.content",
 "it": "\/common\/destiny\_content\/sqlite\/it\/world\_sql\_content\_bd613d541a690aa21896eb4ce761e42f.content",
 "ja": "\/common\/destiny\_content\/sqlite\/ja\/world\_sql\_content\_884749c1a137aa80b701fb175eb9620d.content",
 "pt-br": "\/common\/destiny\_content\/sqlite\/pt-br\/world\_sql\_content\_7a0cc97e802c88dee6a0ecff953194f2.content"
 },
 "mobileGearCDN": {
 "Geometry": "\/common\/destiny\_content\/geometry\/platform\/mobile\/geometry",
 "Texture": "\/common\/destiny\_content\/geometry\/platform\/mobile\/textures",
 "PlateRegion": "\/common\/destiny\_content\/geometry\/platform\/mobile\/plated\_textures",
 "Gear": "\/common\/destiny\_content\/geometry\/gear"
 }
 },
 "ErrorCode": 1,
 "ThrottleSeconds": 0,
 "ErrorStatus": "Success",
 "Message": "Ok",
 "MessageData": {

 }
}

```

### Extracting Information

The main files you will probably want to look at are the "mobileWorldContentPaths" (the other 2 I believe are for getting assets needed to render 3D models). Each of these files are an SQLite database file that has been put into a Zip file.

Extracting them in PHP is pretty straightforward (get\_url is a function I wrote that basically does a cURL request which includes stuff like cookies and my API key etc, file\_get\_contents can be used here, but it can be much slower than using cURL):

```
$cacheFilePath = 'cache/'.pathinfo($path, PATHINFO\_BASENAME);
file\_put\_contents($cacheFilePath.'.zip', get\_url($path));

$zip = new ZipArchive();
if ($zip->open($cacheFilePath.'.zip') === TRUE) {
    $zip->extractTo($\_SERVER['DOCUMENT\_ROOT'].'/cache');
    $zip->close();
}

```

Now if you are like me, and don't have an SQLite viewer on hand, you have an SQLite file that you don't know what tables it contains or what columns are in said tables. Basically each table will generally have two columns, "id" or "key" and "json". Most of the time, "id" is not the entry's hash so you can't just query for a given entry like you can the /Manifest/{type}/{hash} API. So instead, we need to extract these SQLite files further into something we can actually look at and be able to access.

Note: $dbtype is referring to the section in the manifest the file is from (gear, asset, world).

```
if ($db = new SQLite3($dbfile)) {
    $result = $db->query("SELECT name FROM sqlite\_master WHERE type='table'");
    while($row = $result->fetchArray()) {
        $result2 = $db->query('SELECT \* FROM '.$row['name']);
        $data = array();
        while ($col = $result2->fetchArray(true)) {
            $json = json\_decode($col['json'], true);

            if (isset($col['id'])) {
                $data[$col['id']] = $json;
            } else if (isset($col['key'])) {
                $data[$col['key']] = $json;
            } else {
                $data[] = $json;
            }
        }
        if (!file\_exists($dbtype)) mkdir($dbtype);
        file\_put\_contents($dbtype.'/'.$row['name'].'.json', json\_encode($data));
    }
}

```

You should now have some json files you can open up and view. When loading these json files, you should load them in so that the {dataType}Hash value becomes the key so you can reference them directly, though not all tables use a hash to identify objects.

### Entry IDs

It has been discovered that the ID column is actually the hash as an unsigned 32bit number. As long as you properly convert it you can do quick look up queries by ID.

PHP will read it as a signed number so a little conversion is necessary.

```
$id = sprintf('%u', $index & 0xFFFFFFFF);

```

Or if you want to query the ID in your SQLite queries.

```
SELECT CASE WHEN id < 0 THEN id + 4294967296 ELSE id END AS id, json FROM DestinyInventoryItemDefinition

```

## Python

Below is an example from xIntangible's pastebin , describing how to use the data in the manifest in Python.

### Downloading and Extracting the Manifest File

First we get the location of the manifest file, download it, and extract it.

```
 def get\_manifest():
     manifest\_url = 'http://www.bungie.net/Platform/Destiny/Manifest/'

     #get the manifest location from the json
     r = requests.get(manifest\_url)
     manifest = r.json()
     mani\_url = 'http://www.bungie.net'+manifest['Response']['mobileWorldContentPaths']['en']

     #Download the file, write it to 'MANZIP'
     r = requests.get(mani\_url)
     with open("MANZIP", "wb") as zip:
         zip.write(r.content)
     print "Download Complete!"

     #Extract the file contents, and rename the extracted file
     # to 'Manifest.content'
     with zipfile.ZipFile('MANZIP') as zip:
         name = zip.namelist()
         zip.extractall()
     os.rename(name[0], 'Manifest.content')
     print 'Unzipped!'

```
Note that this renames the .content file to 'Manifest.content'. This is just for simplicity.

### Defining Hash Locations for Each Table.

I didn't find it necessary to use the 'id' column, and instead used the hash from the 'json' column in each table, the only hurdle with this is that the hash is stored under a different key for every table.

So, we create a dictionary with the relevant hash locations for each table.

```
 hash\_dict = {'DestinyActivityDefinition': 'activityHash',
              'DestinyActivityTypeDefinition': 'activityTypeHash',
              'DestinyClassDefinition': 'classHash',
              'DestinyGenderDefinition': 'genderHash',
              'DestinyInventoryBucketDefinition': 'bucketHash',
              'DestinyInventoryItemDefinition': 'itemHash',
              'DestinyProgressionDefinition': 'progressionHash',
              'DestinyRaceDefinition': 'raceHash',
              'DestinyTalentGridDefinition': 'gridHash',
              'DestinyUnlockFlagDefinition': 'flagHash',
              'DestinyHistoricalStatsDefinition': 'statId',
              'DestinyDirectorBookDefinition': 'bookHash',
              'DestinyStatDefinition': 'statHash',
              'DestinySandboxPerkDefinition': 'perkHash',
              'DestinyDestinationDefinition': 'destinationHash',
              'DestinyPlaceDefinition': 'placeHash',
              'DestinyActivityBundleDefinition': 'bundleHash',
              'DestinyStatGroupDefinition': 'statGroupHash',
              'DestinySpecialEventDefinition': 'eventHash',
              'DestinyFactionDefinition': 'factionHash',
              'DestinyVendorCategoryDefinition': 'categoryHash',
              'DestinyEnemyRaceDefinition': 'raceHash',
              'DestinyScriptedSkullDefinition': 'skullHash',
              'DestinyGrimoireCardDefinition': 'cardId'}

```

Note, this is every table I know how to use. So if you were only interested in getting data from Tables you planned on using, you could simply remove the tables you were uninterested in.

For example, if you only wanted data on items, perks, and stats:

```
 hashes = {'DestinyInventoryItemDefinition': 'itemHash',
           'DestinyTalentGridDefinition': 'gridHash',
           'DestinyHistoricalStatsDefinition': 'statId',
           'DestinyStatDefinition': 'statHash',
           'DestinySandboxPerkDefinition': 'perkHash',
           'DestinyStatGroupDefinition': 'statGroupHash',}

```

### Creating a Nested Dictionary

Now, we create a nested dictionary. Structured like so:

```
{ 'TableName' : { hash : {json} }

```

This will make more sense when we get to actually using the dictionary.

```

 def build\_dict(hash\_dict):
     #connect to the manifest
     con = sqlite3.connect('Manifest.content')
     print 'Connected'
     #create a cursor object
     cur = con.cursor()

     all\_data = {}
     #for every table name in the dictionary
     for table\_name in hash\_dict.keys():
         #get a list of all the jsons from the table
         cur.execute('SELECT json from '+table\_name)
         print 'Generating '+table\_name+' dictionary....'

         #this returns a list of tuples: the first item in each tuple is our json
         items = cur.fetchall()

         #create a list of jsons
         item\_jsons = [json.loads(item[0]) for item in items]

         #create a dictionary with the hashes as keys
         #and the jsons as values
         item\_dict = {}
         hash = hash\_dict[table\_name]
         for item in item\_jsons:
             item\_dict[item[hash]] = item

         #add that dictionary to our all\_data using the name of the table
         #as a key.
         all\_data[table\_name] = item\_dict
     print 'Dictionary Generated!'
     return all\_data

```

### Call Our Functions and Optionally, Pickle Our Data

Now, we could just call our functions.

```
 get\_manifest()
 all\_data = build\_dict(hashes)

```

But that would be too easy.

Optionally, you could store your new dictionary in a pickle file, and only run this script whenever:

* You want to.
* There is no existing pickle file, like below.
* Or you could get fancy, and do something based on the age of the existing one.

Create a pickle file, if one doesn't exist:

```
 #check if pickle exists, if not create one.
 if os.path.isfile('manifest.pickle') == False:
     get\_manifest()
     all\_data = build\_dict(weapon\_hashes)
     with open('manifest.pickle', 'wb') as data:
         pickle.dump(all\_data, data)
     print "'manifest.pickle' created!\nDONE!"
 else:
     print 'Pickle Exists'

```

### Using Our Nested Dictionary

Below is a simple script using our dictionary as 'all\_data'. This script displays some basic information about everyone's favorite exotic rocket launcher.

```
 with open('manifest.pickle', 'rb') as data:
     all\_data = pickle.load(data)

 hash = 1274330687
 ghorn = all\_data['DestinyInventoryItemDefinition'][hash]

 print 'Name: '+ghorn['itemName']
 print 'Type: '+ghorn['itemTypeName']
 print 'Tier: '+ghorn['tierTypeName']
 print ghorn['itemDescription']

```
Output:

```
Name: Gjallarhorn
Type: Rocket Launcher
Tier: Exotic
"If there is beauty in destruction, why not also in its delivery?" - Feizel Crux

```

This code in it's entirety can be found HERE