THQ's Homefront 2/Technical Design Document

## Summary

The essential technology we're using in Homefront 2 will be CryEngine and that which we've used to ship Crysis 2. This document serves two main purposes (at least for now):

1. analysis of systems used in Crysis 2 (especially those that we're unfamiliar with)
2. to document differences or planned changes from the tech as used in Crysis 2

## New Technology

### AO Support

* CryEngine currently has no support for large-scale AO
* On previous titles this has been hand edited by artists to effect the lighting they want
* This takes them a significant amount of time, and the resulting lighting is expensive on the GPU
* To improve this, we're adding the ability to bake this in Sandbox
* This feature will require extra memory that C2 + C3 did not need (about 2.5Mb per level, using Alien Vessel as a test level - will be roughly based on the number of polys, not level area)
* We will look into streaming this data if necessary
* The tech is now mostly implemented and we're working on improving speed and build process reliability.

### Breakability

* From C2, we know that having many breakable objects in one area can cause big bandwidth spikes if someone throws a grenade in there
	+ Need bandwidth improvements
	+ Need performance improvements
* This needs to work properly with vehicles
	+ When you drive a car into a fence the car stops (because the client is authoritative about the vehicle and the server about the fence).

### New Glass Rendering + Breakability

* New glass rendering system now in-game and in use.
* The new system looks and performs better than the one shipped in Crysis 2.

### Terrain Improvements

* Support for normal maps on terrain, now implemented.
* Further work on terrain and vegetation is scheduled but not yet fully planned.

### Tools and Asset Pipelines

* Sandbox
	+ Asset tagging
	+ Outsourcing tools
* Characters/Weapons
	+ New character pipeline
* Animation
	+ CryMannequin
* LODs
	+ Charlie's cage building LOD generator

### Sound + Music

* Use FMOD
* Take changes from Crysis 3
* The aim will be to ship using that system

### Dedicated Server Arbitrator

* In order to match console players to dedicated servers, we've created a new server process.

### Screenshots

* The player may be able to take pictures using their smartphone
* Essentially this means we need to support taking screenshots and saving them to disk

### Technical Impact of Game Features

* Smartphone photo taking
	+ We don't have lots of memory to store lots of high res photos
* Droid remote control
	+ Has an impact on streaming system
	+ If the droid can move far enough away from the player to stream in new data, and the connection can be instantly interrupted, data by the player may have streamed out
* Drone IK
	+ This feature is no longer required - the walking drones will have wheels instead.
* Chemical weapons
	+ These may make heavy use of particle effects which are costly on GPU and CPU
	+ We can mitigate this to some extent by reducing quality

## Technical Risk Summary

### Memory Use

Crysis 2 and Crysis 3 have both been very close to running out of memory in single player. To mitigate this, we're trying to match their memory footprint as far as possible and keep track of the memory used by levels with our telemetry tools. We're also looking into saving more memory by streaming characters so they don't have to be kept in memory permanently.

### Multiplayer Performance

Crysis 2 ran at roughly 25fps on PS3 in multiplayer, with 12 players. This was largely GPU and render thread related. We should be able to keep these under control by careful creation of content (keeping draw calls down reduces render thread load, reducing light + shader use reduces GPU load). However, in order to support 32 players and vehicles, we'll likely need to carry out some main thread optimization work.

### AI System

The AI shipped in Crysis 2 (and Crysis 1 Conversion) was not as polished as it could have been, and we're keen to make sure the AI in Homefront 2 is better received. Several programmers are already working on this, and we're making sure it's a priority early on to enable the designers to iterate on their levels with a good working AI.

### Anti-Cheat

Crysis 2 multiplayer shipped with very little anti-cheat protection, which made it more difficult to support the community and keep people playing.

## Single Player

### Memory Use

Crysis 2 levels were generally heavily constrained by their memory use - at least one had to be split so it would fit in console memory limits.

During development of Crysis 3, RnD have done lots of work on improving the streaming system to actively unload and manage the assets being used by the game to avoid hitting the same problems again.  We're considering a change to help artists and designers budget earlier in the level creation progress, but this is beyond the scope of HF2.

**Summary**

* CGFs and Textures are streamed, but we need to manage this memory carefully
* AO will use some memory we didn't have before, but streaming this is a possibility
* Moving towards streaming chrs, should help on levels with lots of cutscenes
* Drones and vehicles will use extra memory (though we don't have dropships, aliens, pinger etc)
* Animation streaming use looks like it might be unbounded
* Streaming/resource creation should reduce peak memory use

### Streaming Systems

* Crysis 2 shipped with the ability to stream:
	+ Geometry
		- Can stream lods in and out independently
* Textures
* Animations
* Sound + Music
* Video

### Crysis 2 Level Sizes (for comparison)

* Central Station - not \*that\* big, has the bridge, underground car park (250m x 150m)
* Alien Vessel - 250m x 250m but \*heavily\* constrained once below street level
* Madison Sq - player  travels > 1km, but area is 450m x 500m and not densely populated (in places)
* Central Park - about 500m x 600m, player travels 1.2km
* Battery Park used 206Mb on startup in C3 memory dumps
	+ This was high for a C2 level
	+ It's really not very far off the PS3 limit
	+ 18Mb of animations streamed during the level
	+ 88Mb of zipped geometry streamed
	+ WORKS OUT at 350kb/s of unzipped geometry
	+ >1Gb of textures streamed

* We've looked into a budget for streamed animations.  At present the memory use of the animation streaming is limited only by the assets requested.  The animation system is quite aggressive about unloading animations which are no longer in use, though, so this isn't a priority for Homefront 2.
* The table above includes some textures that are streamed from a pak file which is loaded into memory.  They're not big but do make the texture streaming throughput look higher than it really is.

### CGF Streaming - what tends to go wrong when it breaks?

As you move around too quickly for the data to stream in, the MemReq goes up above the poolsize (even when it's close, this is normally a bad sign).  Models won't be able to load in yet.  As the pool frees some items up and things load from disk, they'll pop in from nowhere.

I think the MemReq means that the lod would have been wanted in the last 4 seconds.  I don't know if this even remotely matches how things are unloaded (that looks like it's using the last 8 seconds and when over budget).
The CGF streaming system will contribute to fragmentation in whatever heap it's allocating from.  On PS3 this will largely be RSX memory, where fragmentation isn't likely to be a big concern.  On 360 it'll be the main heap.  Following some tests, it seems difficult to produce significant fragmentation due to this in Crysis 2, but it's worth us keeping an eye on - especially since we expect our levels to last longer than C2SP levels did.

### Texture Streaming

Whilst playing C2, texture streaming uses more bandwidth than any other system.  When the engine is started up, a pool is allocated for textures (60Mb in RSX on PS3, 90Mb on 360).  Some textures are kept resident in here, some streamed in and out as they're required.  When texture density is too high where the camera is, the streaming system currently tends to thrash (continually kick textures out and replace them with new ones).
The texture streaming system was shipped in Crysis 2, but work is being done on it to:

* Create assets on threads other than the render thread, saving peak memory allocation.
* Understand and actively manage the lifetime of objects better. The tech used in Crysis 2 would often cause thrashing (loading/unloading the same resources repeatedly).

### Streaming Budgets

* Streaming throughput budgets (in kb/s) are difficult to set.
	+ Streaming throughput depends on disk layout of required files.
	+ There's interplay between different systems (sound/music, textures, cgfs, anims).
* We've created a tool so artists can work out why areas are expensive and fix them:

The x-axis on graph below represents the time through a playthrough of the game - each bar represents one frame.  The y-axis shows the texture demand at that time (location and screenshots are available to make the feature a little easier to use).  The tree-view at the top shows the textures the streaming system required in this frame.

### Auto-testing

* We plan to create a proper auto-testing setup for single player
* We already have a base setup for this that worked through C1C
* The automated system would be based on our existing stress test system + Amble
* Amble is our build scripting library
* Grab new build, run the single player levels with the auto-test data
* We plan to integrate this into our continuous automated testing when we have live level setups to test with.  We've done a proof of concept on the CXP level.

### Load and Save

CryEngine supports a standard serialization system that can load from and save to disk.  The system has been used in the Crysis 1 Conversion and Homefront 2.  It relies on the game team implementing save and load functions properly.  We're considering auto-testing the load/save system by auto generating save games using our stress test system.  These would also be useful for skipping through levels later on during development.

### Character Draw Distances

We've set the character draw distances to be higher than those used in Crysis 2 and 3, and given the designers the responsibility for making sure that budgets (number of characters, draw primitives and polys) aren't exceeded.

## Multi-Player

### 32 Players

* As Crytek UK, we've previously shipped 12 player on console and 16 player on PC
* Crysis Wars shipped with 32 player support on PC (the codebase has changed a lot since then)
	+ Bandwidth use was far too high, but we've optimized this considerably
	+ However, @ 3kbits-1 with 32 players, that's:
	+ 3 \* 31 \* 32 -> 2976 kbits
	+ 3 kbits-1 is probably smaller than we can go easily
	+ It also doesn't include joining/leaving/other things
	+ We could look into lodding updates somehow
	+ **Current target based on numbers from multiplay is 2.5Mbit upstream**
* Breakability
	+ Our breakability on C1C and C2MP:
		- Caused bugs
		- Used lots of bandwidth
		- Was cpu and memory hungry
* Lots of the bugs should be fixed, but the other problems still exist.
	+ A large problem in C2MP was caused by the greenhouse.  To mitigate this we might consider having a breakability budget with a maximum amount of breakable stuff in one area.
* Rendering and animation challenges
	+ C2MP was generally render bound (especially on PS3), either by drawcalls (render thread) or shaders (gpu side).
		- We \*should\* be lighter on the shaders, we have fewer Sci-Fi effects
	+ We should avoid creating game modes that draw all players to one point on the map (for example Crash Site from C2), since this will pull lots of player models to the same point on the map.
* Initial tests show our load is largely on the main thread
	+ Sound @ about 5.5ms per frame
	+ 3DEngine @ about 7.9ms
	+ Particles @ about 10.7ms
	+ Physics @ 4.24ms
	+ Entity @ 4.5ms
	+ System @ 6.8ms
	+ Game @ 5.5ms
	+ Animation @ 3.9ms (DummyPlayers)
	+ Network @ 3.44ms
	+ CryAction @ 3.65ms
	+ Renderer @ 1.14ms
	+ Misc @ 40ms
* Chris Bolte is working on optimizing Particles already.
* He's also working on splitting 3DEngine into tasks that can be parallelized
* Dedicated server optimization
	+ The more instances that can be run on one dedicated server the better.

### Vehicles

* These mostly work as they are from C1 (even networked)
* There are some risky issues to fix
	+ head on collisions in networked games need polishing (they've been considerably improved by our anti-cheat work)
	+ colliding with other background physics objects needs work (also improved by our anti-cheat work)
	+ bandwidth use seems too heavy (especially compared to players), but this isn't a big issue right now - we're within our bandwidth targets
* There are a few handling issues, but these should just take some time and iterations

### Lobby Solution

We'd like to use:

* Steam on PC
* XBox 360
	+ Host the servers behind XLSP, these can be normal Linux boxes if we want
	+ Traffic is routed through the SG and stripped of encryption
	+ Players matchmake using Live (like squads)
	+ The entire squad then joins a dedicated server
	+ This is managed by an Arbitrator process, which is new for Homefront 2.  It is currently undergoing testing in our net-tests to make sure it's doing what we want.  Following this we're planning how to stress test it to make sure it can handle peak load at launch.
* PSN
	+ Lee is looking into whether we can login to PSN and use squad matchmaking as planned on 360.

Choosing a lobby solution and getting it set up early will be important - with no Open Beta, we have to get our implementation right first time.  The more testing time we have, the better our chances.

### Anti-Cheat and Anti-Tamper

Crysis 2 shipped with very little anti-cheat protection and (the PC versoon in particular) suffered badly because of this. Warface and Crysis 3 have some changes that should help us.

**Stronger Pak Encryption**

The pak files in C2 shipped using plain TEA encryption and keys you could find using Google. This will be replaced with an asymmetric encryption system so the keys shipped with the game don't allow players to generate new paks. Doing this should prevent casual hacking by players editing XML configuration files.

Work on this has been carried out for Warface. If it proves to be secure, we'll use those changes on HF2, so we're waiting to see how this goes before taking the changes. They are relatively low risk and can wait until near release, even if changes are required.

**Move work to the Server**
Crysis 2 was written with large parts of the code trusting clients. We plan to move what we can to the server.
**Server Validation**
Where we can't move work to the server, we want the server to be responsible for checking that clients are acting within the rules. The server can then automatically kick players who aren't.
**Aim-bot Detection**

We're implementing some aim-bot detection to catch the more obvious aim-bots. This is likely to be an ongoing process though. As we kick and ban people they'll reduce the thresholds they use for their aimbots.

**General Anti-Cheat Work**

We've removed a list of unused RMI calls that could have caused cheat problems. We've added server-side checks for 11 of the 13 functions we currently plan to, with SvRequestHit and SvHealPlayer remaining. (SvRequestHit is a very important one, we're still working out exactly what we want to do with it.) We've moved the vehicle movement to be server authoritative, which makes it more difficult for clients to cheat, and are considering how to do the same for normal player movement.

**Kill-cam Validation**
The server is sent details of the kill-cam by the killer, which can be checked for suspicious data (flying, jumping from one position to another, killing people who they can't see). This data can potentially be stored and reviewed later using a web interface and a PC with an installed copy of HF2.
**Empower the Live team and Community**

We'll improve the in-game UI for kicking and banning players from servers.

**Make sure we can permanently ban cheats**

This didn't work quite right in C2 - the CD key banning had issues (largely caused by lack of testing of the middleware we used and high demand).

### DLC Infastracture

The infrastructure used to deliver DLC depends on the platforms we ship on, but the in-game setup will be the same as that used in Crysis 2 and 3.

### Data Patching

Crysis 2 had the ability to read an RSS feed on startup and grab data to patch some of the XML files with the game. This was used for things like game balancing, but could occasionally be used to fix bugs in data files. It was limited to XML files, and the file had to be re-downloaded each time the used connected. If this download failed, the player would be unable to matchmake against people with the patch.

Crysis 3 are planning to extend this system to patch other data files (especially Lua files). Since they've also re-written the matchmaking algorithm in Lua, this means we should be able to patch the matchmaking algorithm to improve the user's experience. The system will also download the patch locally and only get a new version as it's released.

## Next-gen Consoles

Currently next-gen consoles look likely to be X86 based, and with a significant memory boost over current-gen. We have the HF2 CXP milestone running on Durango, and hope to look into several areas, especially CPU/GPU speed relative to current-gen. This should give us some idea of what we can achieve on the console, particularly:

* Could we run at 60fps in multiplayer?
* Can we render more draw calls in singleplayer?
* How much time do we have to spend on improving post-processing filters + effects?
* How much memory do we have free when running a basic version of the game?
* How should we balance our art + code time between next + current gen versions?

## Character Analysis

Building characters for CXP has given us set of assets to check against budgets. So far things look promising - the new characters seem to have reduced draw calls without reducing visual quality. However, proper evaluation is waiting on some tools work and will happen after CXP.

### Crysis 2 Cutscenes

* Characters with full body animation and facial animation played through Track View
* Sounds + Facial anims match 1:1
* Track View is used for in-game sections where characters talk to the player
	+ e.g. Tara on the lift after rescuing you
	+ This potentially has implications for
		- things like cutscene budgets (financial ones) - the facial mo-cap/sound stuff has to cover some in-game things too
		- it will also make it more difficult to change these in-game pieces once they've been recorded
* Some bone counts for different characters:

* What happens to character models after a Track View has finished?
	+ we know we switch to AI characters, are these lower render quality?
		- Generally no
	+ why are some rendermeshes deleted?  is it a refcount hitting zero?  if so, is there other data for that character we could delete??
	+ We can't stream chr files in, but we probably could (theoretically) if we put it on the rnd plan
		- This means that chrs have to be sync loaded (which normally means at the start of the level) and kept there until any cutscene with them in
* Memory cost of characters
	+ Allocs from CharacterManger::FetchModel (and internally into SetMesh)
	+ Textures will be streamed in and out, so these are basically a question of balance with environment + quality
	+ There's some overhead per character too, but this is difficult to measure and relatively small (looks like ~100kb)
* C2 makes quite heavy use of video clips in cutscenes (things like the Prophet flashbacks are all done like this, lots of TV screens with people talking)
	+ It means you can instantly switch location without streaming issues
	+ It means you can use a pile of models/textures/anims that aren't actually loaded at the moment
	+ These are still rendered in-engine (but offline) and then compressed
* During a playthrough of half of Prism, character memory use settled at about 9.5Mb (peaks at 13.5Mb during loading)
	+ of 9.5Mb:
		- 4Mb is rendermeshes
		- 1.3Mb is ChrParams (anim headers mostly)
		- 1.5Mb is bones
		- 1.2Mb is CAF files
		- 1.1Mb is AIM files
		- leaving about 400kb of miscellaneous stuff (physics proxies, CDF/CGA files etc)
	+ Textures are streamed into texture memory and a bit more tricky to measure...

### Videos

* Videos typically have a soundtrack for non-localised audio
* Variable quality/disk space used, but:
	+ Intro video is about 0.75Mb a second
	+ Battery Park flashback (Prophet shoots himself) is about 0.62Mb/s
	+ Credits movie is about 1.24 Mb/s
* Videos in total used 1.7Gb of disk space
	+ The load screen videos use about 760Mb of disk space
	+ Flashback videos were 150Mb
	+ Tutorial videos were about 140Mb
	+ Other story videos were about 540Mb
	+ Cinematic HUD + logos + misc other videos are ~100Mb

### C2 Cutscene Analysis

Crysis 2 cutscenes are largely (if not entirely) first person. Cuts are rare (which is good, because the engine often deals with these by synchronous stalling until resources for the new location are loaded), and normally dealt with using videos.

**Cutscene from start of "Masks Off"/Prism**

Features 6 characters + several video clips (3m30s total length):

Of note is the light usage of the animations (animations.pak and caf.pak) relative to the sound, music, video and texture load.

**Cutscene from "In at the Deep End"/Intro**

Features 6 characters only, no video clips (1m40s length):

Notice the higher animation load this time, but it's still not that high (2.6Mb) compared to textures and sound effects. The higher load on textures is because of cuts between different cameras (submarine exterior -> interior). I think this leads to low res textures streaming in, but the use of the player being drunk/coming to means you can't see this.

(The animation streaming works out at about 4.5 kB per second per character. This is probably on the low side for 6 active characters, since most of them aren't doing a lot, but it shouldn't be too far from realistic.)

### Part merging

* This task would be to make it easier to produce variations on a character
* It will take a character built in parts and merge them together offline to produce a C2 style character
* This will mean they should be identical to C2 characters from a performance and memory perspective
	+ This is good, but it will mean that 2 virtually identical guys use twice the memory!

## Budgets

### Rendering Budgets

We set render budgets for the maximum cost of a scene in 3 different ways - Polys, Draw Primitives and GPU timings. These are all largely based on those used by C2 and C3.

**Draw Call and Poly budgets**

The drawcall and triangle cost largely affect time on the render thread:

I've grouped the brushes, terrain and vegetation together because we're likely to have large differences between (for example) City Hall and Comms Dish - the exact balance is left up to the background artist. The entities count should leave room for around 12 characters with equipment. In some areas (where we have more vehicles or drones), fewer characters may need to be used to compensate. Designers and artists are free to trade entity draw calls for background ones in areas which are particularly important.

**GPU Budgets**

We set budgets for the GPU separately from the draw calls and rendering, as things like overdraw and shaders can increase the GPU time independently of mesh complexity.

If the game is to run at 30fps (our target), we have 33ms of GPU time every frame.

**33ms** - Whole frame at 30 FPS

* **16.5ms** - Reserved. Code, fixed costs, etc. Empty scene overhead.
* **5ms** - Dynamic/action
* **11.5ms** - Background/static/non-game art

Setting up and empty scene in CryEngine (on PS3) takes about 16.5 milliseconds, and this time is fixed no matter what is drawn. This leaves about the same length of time again for the content.

Crysis 3 MP has budgeted 5ms for dynamic cost (particles, characters, vehicles, etcetera) and 11.5 ms for the background art. The special effects used for their characters (cloak + armour) are likely to be more expensive than those in HF2. However, we're likely to have more drones, vehicles, and characters in general. The chemical weapons could potentially increase particle effect use and cost. Given this it seems to make sense for HF2 to use the same budgets as C3.

These budgets are set for PS3, as this has the weakest GPU of our target platforms. This means that we should measure performance on the PS3.

A quick breakdown of how this time might be split based on some suggested scenes:

These budgets are flexible as long as the total cost is less than or equal to 11.5ms.

### Memory Budgets

On C2MP we had background memory budgets of 55Mb. However, we've turned on static mesh streaming for HF2 multiplayer, which means we now have a pool of 24Mb available but assets will be loaded and freed on the fly.

This is both a good and a bad thing. It means we don't need to concentrate on exactly how much memory the background is using, but makes it more difficult to highlight where things will go wrong.

**Streaming Considerations**

Currently, when the camera jumps more than 16 metres between consecutive frames, the engine will synchronously load the required assets (causing a stall). When using our MP test level and DVD emulation on 360, this sync stall typically takes about a second to read 1.5Mb or so (this stall is too long for release - around 0.5 sec might be okay when spawning or switching drone).

Such camera jumps typically occur in multiplayer when respawning, and potentially also when switching from a remote control drone back to the player. Fixing this sync stall will likely require one of:

* some code on spawning to hint about the spawn point that's been chosen and streaming code to take two camera positions into account
* we could try sync loading only the lowest lods + streaming the rest
* we could keep lowest lods for everything in memory (setting a memory budget for this and then reducing the streaming pool size to make up) and sync stall on (or stream) the rest

### Texture and Streaming Budgets

The texture streaming system has a fixed size pool (at least 60Mb, which is on PS3). This has to contain textures that can't be streamed (hud textures, for example) as well as all the textures needed to render the environment and characters. Beyond this, we haven't set specific budgets for texture use in previous titles.

We have the ability to stream in and out different levels of mipmaps. For C2, we held a pak containing low-level textures permanently in memory (whilst in level) to reduce the streaming load of these textures (and initial load time). Generally this means that when the texture streaming system is overloaded, it manifests as low res textures and thrashing (as well as an on screen warning), and artists attempt to reduce the texture use at this point.

We have some heatmap tools in Sandbox for tracking down expensive areas in a level at a glance.

**Homefront 2 Specific Features**

The Smartphone UI and Screenshots have texture budget impacts. If we store screenshots for use later, these will need to be in texture memory somewhere - they can't be swapped out and streamed back in because some target hardware won't have a HDD (360 Arcade).

### Design and AI Budgets

AI characters are expensive on the CPU, as well as from a rendering side, and it's important to make sure we don't have too many active at one time. They are currently limited to 16 entities using the Entity Pool system.

### Disk Space Budgets

Our target disk space budget is likely to be a single Xbox 360 disk (7.8 Gb).

Crysis 2 used 5.9Gb for 2 languages:

CPU, GPU, Memory, AI

Artwork, animation, sound, design

### Budget Testing

We can automatically test most of our budgets using our automatic testing systems. The system runs several different types of tests:

* Flythroughs - a designer creates a route and checks it into Perforce. The automatic testing system then runs the game and flies a camera around the route 6 times, once facing north, south, east, west, up and down.
* Auto-testing - a designer or coder sets up a script that plays through Single Player levels automatically, roughly following the intended path the end user will take.
* Stress-testing - the game runs the level and throws 16 dummy players in there for a set period of time before moving on to the next level.

These all plugin to the same scripting and reporting system that e-mails the team when tests fail and that reports telemetry to an internal webserver. This telemetry includes the number of draw calls, number of triangles, memory use, and a host of other stats. These stats can be quickly checked against the budgets above. We don't currently have a system to do this automatically, but we'll look to add one when we think it would be useful.

## Loading Times

When the levels are settled at little more, we will fix up the build system so we have fastload and streaming paks generated by the build process (level cache paks have been created). The process has been used in Crysis 2 and 3, but adds time to the build process, especially when creating lots of new levels.

### Frontend load times

For Crysis 2 loading time reduction, we make:

* \_FastLoad/IntroMovies.pak
* \_FastLoad/startup.pak

Which speed up loading into the frontend from cold boot. (Startup.pak contains the config files needed to load into the frontend, and keeps them all coherent on disk).

Adding these paks to the build process should be enough to reduce loading times to within TRC constraints again.

### Level load times

The \_LevelCache folder keeps the small files needed on level load together in one place on disk to reduce seek times.

## Destructibility

We've created a simpler path for cheaper breakable assets that should be easier for artists to work with. Frankfurt are working on a destructibility editor we're keeping an eye on as well, this might be useful for more complicated objects; it may be beyond the scope of Homefront 2 though.

## Entity Pool System

An Entity Pool is a collection of slots which a given type of entity can be prepared from. It's used in CryEngine to limit the number of a given type of entity that can exist in memory. When an entity is needed during gameplay it is Prepared from the pool, and when it's no longer needed, it's Freed back again.

There is a hard limit to the number of entities that can exist in the pool - for Crysis 2 this number is 16 for the AI. When the game attempts to create a 17th AI, it will fail.

For HF2, we've added the ability to budget across different classes (AI vs Vehicles vs Drones).

## Appendix - Collecting Information from CryEngine

### Streaming

* Copy a 360 build to your PC HDD.
* Run the StartDVDEmu.bat file in that folder.
* Enter a logfile name.
* Run the dvd:\Homefront2.exe on your 360 kit.
* use sys\_streaming\_debug 4 to get a streaming log to xe:\\dvd\TestResults\StreamingLog.txt.