Keras.io Tensorflow Neural Network Out of Memory on GPU Issue
$25-50 USD / hour
กำลังดำเนินการ
โพสต์ เกือบ 6 ปีที่ผ่านมา
$25-50 USD / hour
I am currently recieving an OOM Error while using GridSearchCV function and grid.fit. The issue is I only recieve the OOM error when my SECOND fit starts...
I've tried turning Batch Size from 100 down to 1, and even only 1 epoch still same issue.
The exact error is:
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[12288,12288] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Anyone have any idea to better handle the OOM issue other than "smaller dataset" ?
I'm running GTX 1080 32GB RAM and Tensorflow is set to use GPU
Not sure if I provided enough information, let me know if I'm missing anything that would help diagnose.
--------
My settings are as follows:
def create_model(optimizer=Adam, learn_rate=0.001):
inputs = Input(shape=(image_width*image_height*3,))
outputs = Dense(image_width*image_height*3, activation='tanh' )(inputs)
model = Model(inputs=inputs, outputs=outputs)
model_optimizer = optimizer(lr=learn_rate)
[login to view URL](optimizer=model_optimizer, loss='mse', metrics=['mae'])
return model
model = KerasRegressor(build_fn=create_model, verbose=1)
learn_rate_list = [0.03]
batch_size = [1]
optimizer_list = [Adam]
epochs = [1]
param_grid = dict(optimizer=optimizer_list, learn_rate=learn_rate_list, epochs=epochs, batch_size=batch_size)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1,
scoring='neg_mean_absolute_error', verbose=1, cv=2)
I will try to earn my 2 cents with one shot try.
In past, I had similar issue. It was kind of different because I was running on CPU, but it was also memory issue. My Process did grew up to 70+ GB of RAM instead of normal 2-3 GB.
It was also hunger for memory in second, third and next fit iterations. Then I did fix it by using some memory cleanup from step to step. I did train network from scratch with a different hyper-parameters, so you should probably retain Tensorflow graph. That said, you might need to disable 'tf.reset_default_graph()'.
def cleanup():
import keras.backend.tensorflow_backend
if keras.backend.tensorflow_backend._SESSION:
import tensorflow as tf
tf.reset_default_graph()
[login to view URL]()
keras.backend.tensorflow_backend._SESSION = None
from keras import backend as be
be.clear_session()
# [login to view URL]()
$25 USD ใน 5 วัน
5.0 (2 รีวิว)
2.7
2.7
5 freelancers are bidding on average $35 USD/ชั่วโมง for this job
Hi
I am a qualified python developer with rich experiences of machine learning. I am interesting in this project and am able to help you. I am ready to start the work.
Looking forward to hearing from you soon.
Best Regards,
Yongtao
I have a good hands on working with Advanced Excel, R and Python and BI tools and technologies, AI, Big Data. I have quite a good knowledge of DL/ML Algorithm , have also developed Dashboards and Web Application. My area of expertise is building financial models (Stock Markets) , Image Processing and building models for food, healthcare and telecom sector, Classification/Prediction/Clustering, NLP and Chatbots.
I understand the project requirement and will deliver the desired product within the time specified.
I would like to hear from you.
Thanks
Shivam