Heap Memory Usage

Introduction

The heap is a critical memory area where Neo4j stores temporary data during query execution and transaction processing.

In this lesson, you’ll learn how to monitor heap memory usage and identify when memory pressure is affecting your performance.

Understanding Heap Memory

Heap memory is where Neo4j stores data during query execution and transaction processing. When you execute a read query, intermediate results accumulate in the heap until they’re ready to stream back to your application. Complex queries with many operators consume more heap memory than simple queries.

When you write data to Neo4j, the database holds changes in memory to maintain ACID guarantees. The entire unit of work stays in the heap until it’s complete and ready to write to the transaction log. This prevents transactions from accumulating too many changes in memory before commit.

The heap also stores transaction state, cached query plans, and internal data structures. The memory you need depends on your query complexity, number of concurrent queries, transaction size, and working set size.

ACID Compliance

Neo4j is an ACID compliant database. This is a key feature that helps databases maintain data integrity and consistency.

The acronym stands for:

Atomicity - All operations in a transaction succeed or every operation is rolled back.
Consistency - On the completion of a transaction, the database is structurally sound.
Isolation - Transactions do not contend with one another. Contentious access to data is moderated by the database so that transactions appear to run sequentially.
Durability - The results of applying a transaction are permanent, even in the presence of failures.

Monitoring Heap Memory

The Heap Memory metric shows the percentage of configured heap memory in use. The expected baseline is 20-50% during regular operations. Temporary spikes to 60-80% during complex queries are normal and healthy.

The key indicator of problems is sustained high usage above 80%. Short spikes are acceptable, but if heap usage stays above 80% for extended periods, you have memory pressure that will cause increased garbage collection, slower queries, and risk of Out of Memory errors.

Understanding heap spikes

Occasional spikes to 70-80% are normal during complex queries. The concern is sustained high usage, not temporary peaks.

Mitigating Heap Memory Issues

When you see sustained heap usage above 80%, start by optimizing your queries. Use LIMIT clauses to avoid returning large result sets. Break complex queries into smaller operations. Use CALL { } IN TRANSACTIONS OF 1000 ROWS for batch operations to process data in smaller chunks.

Transaction management is equally important. Keep transactions short-lived and commit them promptly. Long-running transactions hold memory and prevent the garbage collector from reclaiming space. Stream results when possible using your Neo4j driver instead of collecting all results in memory.

When to Scale Your Instance

After optimizing your queries and transactions, you may still see sustained heap usage above 80%. This indicates your workload legitimately requires more memory than your current instance provides. Scaling is the right choice when optimization efforts have been exhausted.

Scale your instance when heap usage remains consistently above 80% despite query optimization. You should also scale if you’ve optimized queries but they still fail with Out of Memory errors during normal operations. Performance degradation that persists after optimization is another clear indicator.

However, always optimize first before scaling. Inefficient queries will consume any amount of memory you provide. If you scale without fixing query issues, you’ll only delay the problem and waste resources.

Optimizing queries

Use query optimization tools like EXPLAIN and PROFILE to verify your queries are efficient before scaling. Any expensive operations are highlighted in orange on the query plan.

Check Your Understanding

Heap Memory Threshold

What does sustained heap memory usage above 80% indicate?

❏ Normal operation for complex workloads
❏ The garbage collector is working efficiently
✓ Memory pressure requiring optimization or scaling
❏ Temporary spike that will resolve automatically

Hint

Consider what the expected baseline is and when sustained high usage becomes a problem.

Solution

Memory pressure requiring optimization or scaling is correct.

Sustained heap usage above 80% indicates memory pressure that causes increased garbage collection, slower queries, and risk of Out of Memory errors. The expected baseline is 20-50% with temporary spikes to 60-80%.

Normal operation is incorrect because sustained high usage is not normal. Efficient garbage collector is wrong because high heap usage increases garbage collection frequency. Temporary spike is incorrect because sustained means it persists, not resolves automatically.

Optimizing Heap Usage

What is the recommended first step when heap usage is consistently above 80%?

❏ Immediately scale the instance to a larger size
✓ Optimize queries and transaction management
❏ Restart the instance to clear memory
❏ Reduce the number of concurrent connections

Hint

Inefficient queries will consume memory regardless of instance size.

Solution

The correct course of action is to Optimize queries and transaction management.

Always investigate and optimize your query patterns before scaling, as inefficient queries will eventually consume any amount of memory. Use LIMIT clauses, batch operations, and keep transactions short-lived to reduce heap memory usage.

It would be premature to scale the instance without optimization. Restarting the instance only temporarily clears memory but doesn’t fix the underlying cause. Reducing connections doesn’t address query or transaction issues consuming heap memory.

Summary

The Heap Memory metric shows the percentage of configured heap memory in use. Expected baseline is 20-50% with temporary spikes to 60-80% during complex queries. Sustained usage above 80% indicates memory pressure that requires optimization or scaling.

Optimize queries first using LIMIT clauses and batch operations. Keep transactions short-lived. Scale your instance only after optimization if heap usage remains consistently high.

In the next lesson, you’ll learn about garbage collection and how excessive heap usage impacts it.

Aura In Production

Backup and Restore

Logs and Query Optimization

Monitoring Resources

Monitoring Instance Performance

Monitoring Database Health

Advanced Monitoring