Neural networks are memory hungry and more ambitious SKIL projects will require more memory allocated to SKIL and Notebooks. Often a lack of memory appears as a
java.lang.OutOfMemoryError and knowing how to change your memory settings can prevent this in the future.
To fix memory issues in Zeppelin (the system behind Notebooks), increase the amount of JVM RAM available. Zeppelin's default configuration provides 8 GB of RAM on the JVM, and 4 GB of off-heap space. You can expand these by changing the
DEFAULT_ZEPPELIN_JVM_ARGS environment variable and deleting the Zeppelin interpreter process in the Processes tab.
There are a couple ways to set this variable:
- Specify when launching SKIL Docker container with
- Add it to
/etc/skil/skil-env.sh(or /etc/profile.d/skil-env.sh if /etc/skil-env.sh doesn't exist)
If you have a GPU such as a K80, the following lines are enough to maximize your system:
DEFAULT_ZEPPELIN_JVM_ARGS=-Xmx16g -Dorg.bytedeco.javacpp.maxbytes=16G -Dorg.bytedeco.javacpp.maxphysicalbytes=16G -Dorg.nd4j.versioncheck=false -Dorg.deeplearning4j.config.custom.enabled=false
Once you have updated your settings, you will need to restart SKIL. If you have a bare-metal SKIL installation:
sudo systemctl stop skil sudo systemctl start skil
Otherwise you'll need to
docker exec into the SKIL Docker instance and restart it from a bash instance.
In certain cases you may need to ensure Docker has enough memory at runtime in addition to other adjustments. Docker on Windows and OSX have these features built-in to the user interface. If running on Linux or a server and using the Docker CLI, you can pass a value to the
--memory flag. Note that this is typically used for limiting memory.
You can read more on the Docker website.
Often when a SKIL workspace or experiment is constrained by memory, it will appear as something similar as:
java.lang.OutOfMemoryError: Cannot allocate new DoublePointer(100000000): totalBytes = 588M, physicalBytes = 4G at org.bytedeco.javacpp.DoublePointer.<init>(DoublePointer.java:76) at org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:549) ...
Depending on the data type of NDArray used in your code, this can also appear with
FloatPointers or occur during memory deallocation.