Notebook

Offline reinforcement learning with Ray AIR¶

In this example, we'll train a reinforcement learning agent using offline training.

Offline training means that the data from the environment (and the actions performed by the agent) have been stored on disk. In contrast, online training samples experiences live by interacting with the environment.

Let's start with installing our dependencies:

In [1]:

# !pip install -qU "ray[rllib]" gymnasium

Now we can run some imports:

In [2]:

import gymnasium as gym

import numpy as np
import ray
from ray.air import Checkpoint
from ray.air.config import CheckpointConfig, RunConfig
from ray.train.rl.rl_predictor import RLPredictor
from ray.train.rl.rl_trainer import RLTrainer
from ray.air.config import ScalingConfig
from ray.air.result import Result
from ray.rllib.algorithms.bc import BC
from ray.tune.tuner import Tuner

/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:18: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  DESCRIPTOR = _descriptor.FileDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:36: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:43: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:29: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _TENSORSHAPEPROTO_DIM = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:73: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:80: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:66: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _TENSORSHAPEPROTO = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:19: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  DESCRIPTOR = _descriptor.FileDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:33: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:37: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:41: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:45: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:49: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:53: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:57: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:61: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:65: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:69: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:73: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:77: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:81: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:85: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:89: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:93: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:97: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:101: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:105: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:109: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:113: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:117: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:121: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:125: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:129: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:133: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:137: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:141: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:145: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:149: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:153: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:157: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:161: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:165: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:169: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:173: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:177: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:181: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:185: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:189: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:193: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:197: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:201: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:205: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:209: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:213: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:217: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:27: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _DATATYPE = _descriptor.EnumDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:287: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:280: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _SERIALIZEDDTYPE = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:20: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  DESCRIPTOR = _descriptor.FileDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:39: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:46: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:32: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _RESOURCEHANDLEPROTO_DTYPEANDSHAPE = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:76: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:83: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:90: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:97: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:104: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:111: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:69: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _RESOURCEHANDLEPROTO = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  DESCRIPTOR = _descriptor.FileDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:40: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:47: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:54: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:61: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:68: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:75: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:89: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:96: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:103: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:110: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:117: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:124: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:131: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:138: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:145: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:152: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:33: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _TENSORPROTO = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:183: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:190: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:197: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:176: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _VARIANTTENSORDATAPROTO = _descriptor.Descriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  DESCRIPTOR = _descriptor.FileDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:40: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:47: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:54: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:61: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:68: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:75: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.FieldDescriptor(
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:61: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if (distutils.version.LooseVersion(tf.__version__) <
/Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:62: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  distutils.version.LooseVersion(required_tensorflow_version)):
WARNING:tensorflow:From /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py:561: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version.
Instructions for updating:
experimental_relax_shapes is deprecated, use reduce_retracing instead

We will be training on offline data - this means we have full agent trajectories stored somewhere on disk and want to train on these past experiences.

Usually this data could come from external systems, or a database of historical data. But for this example, we'll generate some offline data ourselves and store it using RLlibs output_config.

In [3]:

def generate_offline_data(path: str):
    print(f"Generating offline data for training at {path}")
    trainer = RLTrainer(
        algorithm="PPO",
        run_config=RunConfig(stop={"timesteps_total": 5000}),
        config={
            "env": "CartPole-v1",
            "output": "dataset",
            "output_config": {
                "format": "json",
                "path": path,
                "max_num_samples_per_file": 1,
            },
            "batch_mode": "complete_episodes",
            "framework": "torch"
        },
    )
    trainer.fit()

Here we define the training function. It will create an RLTrainer using the PPO algorithm and kick off training on the CartPole-v1 environment. It will use the offline data provided in path for this.

In [4]:

def train_rl_bc_offline(path: str, num_workers: int, use_gpu: bool = False) -> Result:
    print("Starting offline training")
    dataset = ray.data.read_json(
        path, parallelism=num_workers, ray_remote_args={"num_cpus": 1}
    )

    trainer = RLTrainer(
        run_config=RunConfig(
            stop={"training_iteration": 5},
            checkpoint_config=CheckpointConfig(checkpoint_at_end=True)
        ),
        scaling_config=ScalingConfig(num_workers=num_workers, use_gpu=use_gpu),
        datasets={"train": dataset},
        algorithm=BC,
        config={
            "env": "CartPole-v1",
            "framework": "tf",
            "evaluation_num_workers": 1,
            "evaluation_interval": 1,
            "evaluation_config": {"input": "sampler"},
            "framework": "torch"
        },
    )
    result = trainer.fit()
    return result

Once we trained our RL policy, we want to evaluate it on a fresh environment. For this, we will also define a utility function:

In [5]:

def evaluate_using_checkpoint(checkpoint: Checkpoint, num_episodes) -> list:
    predictor = RLPredictor.from_checkpoint(checkpoint)

    env = gym.make("CartPole-v1")

    rewards = []
    for i in range(num_episodes):
        obs, _ = env.reset()
        reward = 0.0
        terminated = truncated = False
        while not terminated and not truncated:
            action = predictor.predict(np.array([obs]))
            obs, r, terminated, truncated, _ = env.step(action[0])
            reward += r
        rewards.append(reward)

    return rewards

Let's put it all together. First, we initialize Ray and create the offline data:

In [10]:

# ray.init(num_cpus=8)

path = "/tmp/out"
generate_offline_data(path)

Generating offline data for training at /tmp/out

Tune Status

Current time:	2023-03-30 10:01:06
Running for:	00:00:16.15
Memory:	18.1/32.0 GiB

System Info

Using FIFO scheduling algorithm.
Logical resource usage: 0/8 CPUs, 0/0 GPUs

Trial Status

Trial name	status	loc	iter	total time (s)	ts	reward	episode_reward_max	episode_reward_min	episode_len_mean
AIRPPO_6c451_00000	TERMINATED	127.0.0.1:82000	2	7.14829	8179	43.54	143	10	43.54

(pid=82000) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 5x across cluster]
(pid=82000)   DESCRIPTOR = _descriptor.FileDescriptor( [repeated 5x across cluster]
(pid=82000) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 40x across cluster]
(pid=82000)   _descriptor.FieldDescriptor( [repeated 40x across cluster]
(pid=82000) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:176: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 7x across cluster]
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:217: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 47x across cluster]
(pid=82000)   _descriptor.EnumValueDescriptor( [repeated 47x across cluster]
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(pid=82000)   _descriptor.FieldDescriptor(
(ReadJSON pid=81351) 2023-03-30 09:58:43,544	INFO worker.py:837 -- Task failed with retryable exception: TaskID(85bd3840f615b0e4ffffffffffffffffffffffff01000000). [repeated 7x across cluster]
(ReadJSON pid=81351) Traceback (most recent call last): [repeated 7x across cluster]
(ReadJSON pid=81351)   File "python/ray/_raylet.pyx", line 651, in ray._raylet.execute_dynamic_generator_and_store_task_outputs [repeated 7x across cluster]
(ReadJSON pid=81351)   File "python/ray/_raylet.pyx", line 2533, in ray._raylet.CoreWorker.store_task_outputs [repeated 7x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/execution/operators/map_operator.py", line 374, in _map_task [repeated 8x across cluster]
(AIRBC pid=81380)     for b_out in fn(iter(blocks), ctx): [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/execution/legacy_compat.py", line 287, in do_map [repeated 8x across cluster]
(AIRBC pid=81380)     yield from block_fn(blocks, ctx, *fn_args, **fn_kwargs) [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/plan.py", line 1317, in wrapper [repeated 8x across cluster]
(AIRBC pid=81380)     yield from fn(block, ctx, *args, **kwargs) [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/plan.py", line 1195, in block_fn [repeated 8x across cluster]
(AIRBC pid=81380)     for block in read_fn(): [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/datasource/file_based_datasource.py", line 500, in read_files [repeated 8x across cluster]
(AIRBC pid=81380)     yield output_buffer.next() [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/output_buffer.py", line 74, in next [repeated 8x across cluster]
(AIRBC pid=81380)     block = self._buffer.build() [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/delegating_block_builder.py", line 70, in build [repeated 8x across cluster]
(AIRBC pid=81380)     return self._builder.build() [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/table_block.py", line 118, in build [repeated 8x across cluster]
(AIRBC pid=81380)     return self._concat_tables(tables) [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/arrow_block.py", line 127, in _concat_tables [repeated 8x across cluster]
(AIRBC pid=81380)     return transform_pyarrow.concat(tables) [repeated 8x across cluster]
(AIRBC pid=81380)   File "/Users/avnish/ray/python/ray/data/_internal/arrow_ops/transform_pyarrow.py", line 255, in concat [repeated 8x across cluster]
(AIRBC pid=81380)     table = pyarrow.concat_tables(blocks, promote=True) [repeated 8x across cluster]
(AIRBC pid=81380)   File "pyarrow/table.pxi", line 2338, in pyarrow.lib.concat_tables [repeated 8x across cluster]
(AIRBC pid=81380)   File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status [repeated 8x across cluster]
(AIRBC pid=81380)   File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status [repeated 8x across cluster]
(AIRBC pid=81380) pyarrow.lib.ArrowInvalid: Unable to merge: Field policy_batches has incompatible types: struct<default_policy: struct<action_dist_inputs: list<item: list<item: double>>, action_logp: list<item: double>, action_prob: list<item: double>, actions: list<item: int64>, advantages: list<item: double>, agent_index: list<item: int64>, eps_id: list<item: int64>, new_obs: string, obs: string, prev_actions: list<item: int64>, prev_rewards: list<item: double>, rewards: list<item: double>, t: list<item: int64>, terminateds: list<item: bool>, truncateds: list<item: bool>, unroll_id: list<item: int64>, value_targets: list<item: double>, vf_preds: list<item: double>>> vs struct<default_policy: struct<action_dist_inputs: list<item: list<item: double>>, action_logp: list<item: double>, action_prob: list<item: double>, actions: list<item: int64>, advantages: list<item: double>, agent_index: list<item: int64>, eps_id: list<item: int64>, infos: list<item: struct<0: struct<agent0: struct<>>>>, new_obs: string, obs: string, prev_actions: list<item: int64>, prev_rewards: list<item: double>, rewards: list<item: double>, t: list<item: int64>, terminateds: list<item: bool>, truncateds: list<item: bool>, unroll_id: list<item: int64>, value_targets: list<item: double>, vf_preds: list<item: double>>> [repeated 8x across cluster]
(pid=82000) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:61: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
(pid=82000)   if (distutils.version.LooseVersion(tf.__version__) <
(pid=82000) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:62: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
(pid=82000)   distutils.version.LooseVersion(required_tensorflow_version)):
(pid=82000) WARNING:tensorflow:From /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py:561: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version.
(pid=82000) Instructions for updating:
(pid=82000) experimental_relax_shapes is deprecated, use reduce_retracing instead
(AIRPPO pid=82000) 2023-03-30 10:00:54,374	WARNING algorithm_config.py:636 -- Cannot create PPOConfig from given `config_dict`! Property __stdout_file__ not supported.
(AIRPPO pid=82000) 2023-03-30 10:00:54,479	INFO algorithm.py:527 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(pid=82019)   _TENSORSHAPEPROTO_DIM = _descriptor.Descriptor(
(pid=82019)   _TENSORSHAPEPROTO = _descriptor.Descriptor(
(pid=82019) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:27: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82019)   _DATATYPE = _descriptor.EnumDescriptor(
(pid=82019)   _SERIALIZEDDTYPE = _descriptor.Descriptor(
(pid=82019)   _RESOURCEHANDLEPROTO_DTYPEANDSHAPE = _descriptor.Descriptor(
(pid=82019)   _RESOURCEHANDLEPROTO = _descriptor.Descriptor(
(pid=82019)   _TENSORPROTO = _descriptor.Descriptor(
(pid=82019)   _VARIANTTENSORDATAPROTO = _descriptor.Descriptor(
(pid=82020) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 10x across cluster]
(pid=82020)   DESCRIPTOR = _descriptor.FileDescriptor( [repeated 10x across cluster]
(pid=82020) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 80x across cluster]
(pid=82020)   _descriptor.FieldDescriptor( [repeated 80x across cluster]
(pid=82020) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:176: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 14x across cluster]
(pid=82020) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:217: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 94x across cluster]
(pid=82020)   _descriptor.EnumValueDescriptor( [repeated 94x across cluster]
(AIRPPO pid=82000) 2023-03-30 10:00:59,026	WARNING util.py:67 -- Install gputil for GPU system monitoring.
(RolloutWorker pid=82019) 2023-03-30 10:01:00,115	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:62: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. [repeated 4x across cluster]
(pid=82020)   if (distutils.version.LooseVersion(tf.__version__) < [repeated 2x across cluster]
(pid=82020)   distutils.version.LooseVersion(required_tensorflow_version)): [repeated 2x across cluster]
(pid=82020) WARNING:tensorflow:From /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py:561: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version. [repeated 2x across cluster]
(pid=82020) Instructions for updating: [repeated 2x across cluster]
(pid=82020) experimental_relax_shapes is deprecated, use reduce_retracing instead [repeated 2x across cluster]
(pid=82019) Resource usage vs limits 0:   0%|                                                                                                                                                      | 0/1 [00:00<?, ?it/s]
(pid=82019) Repartition 1:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]

(pid=82019) Repartition 2:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]

(pid=82019)   *- Repartition 2:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]


(pid=82019) Write 3:   0%|                                                                                                                                                                         | 0/1 [00:00<?, ?it/s]



(pid=82019) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82019) output: 0 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]
(pid=82019) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1:   0%|                                                                                                                    | 0/1 [00:00<?, ?it/s]


(pid=82019) Write: 0 active, 0 queued, 0.0 MiB objects 3:   0%|                                                                                                                                    | 0/1 [00:00<?, ?it/s]
(pid=82019) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.32 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                    | 0/1 [00:00<?, ?it/s]
(pid=82019) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 130.81it/s]


(pid=82019) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82019) output: 1 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]


                                                                                                                                                                                                                         , 114.09it/s]
                                                                                                                                                                                                                         

                                                                                                                                                                                                                         


                                                                                                                                                                                                                         



(pid=82020) Resource usage vs limits 0:   0%|                                                                                                                                                      | 0/1 [00:00<?, ?it/s]
(pid=82020) Repartition 1:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]

(pid=82020) Repartition 2:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]

(pid=82020)   *- Repartition 2:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]


(pid=82020) Write 3:   0%|                                                                                                                                                                         | 0/1 [00:00<?, ?it/s]



(pid=82020) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82020) output: 0 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]
(pid=82020) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1:   0%|                                                                                                                    | 0/1 [00:00<?, ?it/s]


(pid=82020) Write: 0 active, 0 queued, 0.0 MiB objects 3:   0%|                                                                                                                                    | 0/1 [00:00<?, ?it/s]
(pid=82020) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.32 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                    | 0/1 [00:00<?, ?it/s]
(pid=82020) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 80.83it/s]


(pid=82020) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82020) output: 1 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]


                                                                                                                                                                                                                         0, 82.36it/s]

Trial Progress

Trial name	agent_timesteps_total	connector_metrics	counters	custom_metrics	date	done	episode_len_mean	episode_media	episode_reward_max	episode_reward_mean	episode_reward_min	episodes_this_iter	episodes_total	hostname	info	iterations_since_restore	node_ip	num_agent_steps_sampled	num_agent_steps_trained	num_env_steps_sampled	num_env_steps_sampled_this_iter	num_env_steps_trained	num_env_steps_trained_this_iter	num_faulty_episodes	num_healthy_workers	num_in_flight_async_reqs	num_remote_worker_restarts	num_steps_trained_this_iter	perf	pid	policy_reward_max	policy_reward_mean	policy_reward_min	sampler_perf	sampler_results	time_since_restore	time_this_iter_s	time_total_s	timers	timestamp	timesteps_total	training_iteration	trial_id
AIRPPO_6c451_00000	8179	{'ObsPreprocessorConnector_ms': 0.0020761489868164062, 'StateBufferConnector_ms': 0.0016286373138427734, 'ViewRequirementAgentConnector_ms': 0.0386197566986084}	{'num_env_steps_sampled': 8179, 'num_env_steps_trained': 8179, 'num_agent_steps_sampled': 8179, 'num_agent_steps_trained': 8179}	{}	2023-03-30_10-01-06	True	43.54	{}	143	43.54	10	90	266	avnishs-mbp-3.lan	{'learner': {'default_policy': {'learner_stats': {'allreduce_latency': 0.0, 'grad_gnorm': 1.0311362160602584, 'cur_kl_coeff': 0.29999999999999993, 'cur_lr': 5.0000000000000016e-05, 'total_loss': 8.954141706724961, 'policy_loss': -0.029519098582871568, 'vf_loss': 8.978031371037165, 'vf_explained_var': 0.04644986521452665, 'kl': 0.018764853765244092, 'entropy': 0.6120760556931297, 'entropy_coeff': 0.0}, 'model': {}, 'custom_metrics': {}, 'num_agent_steps_trained': 128.0, 'num_grad_updates_lifetime': 1410.5, 'diff_num_grad_updates_vs_sampler_policy': 479.5}}, 'num_env_steps_sampled': 8179, 'num_env_steps_trained': 8179, 'num_agent_steps_sampled': 8179, 'num_agent_steps_trained': 8179}	2	127.0.0.1	8179	8179	8179	4153	8179	4153	0	2	0	0	4153	{'cpu_util_percent': 29.48, 'ram_util_percent': 56.42}	82000	{}	{}	{}	{'mean_raw_obs_processing_ms': 0.1494050133826096, 'mean_inference_ms': 0.3030270243531414, 'mean_action_processing_ms': 0.04484232997177898, 'mean_env_wait_ms': 0.020527675791925257, 'mean_env_render_ms': 0.0}	{'episode_reward_max': 143.0, 'episode_reward_min': 10.0, 'episode_reward_mean': 43.54, 'episode_len_mean': 43.54, 'episode_media': {}, 'episodes_this_iter': 90, 'policy_reward_min': {}, 'policy_reward_max': {}, 'policy_reward_mean': {}, 'custom_metrics': {}, 'hist_stats': {'episode_reward': [10.0, 21.0, 20.0, 12.0, 15.0, 35.0, 16.0, 32.0, 21.0, 19.0, 72.0, 35.0, 32.0, 102.0, 76.0, 21.0, 15.0, 17.0, 15.0, 43.0, 25.0, 34.0, 16.0, 26.0, 71.0, 28.0, 52.0, 69.0, 10.0, 21.0, 12.0, 130.0, 56.0, 18.0, 35.0, 32.0, 108.0, 61.0, 97.0, 47.0, 31.0, 23.0, 18.0, 20.0, 59.0, 13.0, 69.0, 58.0, 38.0, 22.0, 83.0, 25.0, 22.0, 34.0, 53.0, 29.0, 18.0, 115.0, 25.0, 59.0, 20.0, 28.0, 64.0, 43.0, 40.0, 36.0, 93.0, 13.0, 20.0, 35.0, 29.0, 38.0, 10.0, 58.0, 26.0, 126.0, 17.0, 54.0, 35.0, 55.0, 35.0, 37.0, 110.0, 28.0, 41.0, 31.0, 45.0, 41.0, 20.0, 123.0, 17.0, 136.0, 41.0, 59.0, 51.0, 18.0, 14.0, 98.0, 35.0, 143.0], 'episode_lengths': [10, 21, 20, 12, 15, 35, 16, 32, 21, 19, 72, 35, 32, 102, 76, 21, 15, 17, 15, 43, 25, 34, 16, 26, 71, 28, 52, 69, 10, 21, 12, 130, 56, 18, 35, 32, 108, 61, 97, 47, 31, 23, 18, 20, 59, 13, 69, 58, 38, 22, 83, 25, 22, 34, 53, 29, 18, 115, 25, 59, 20, 28, 64, 43, 40, 36, 93, 13, 20, 35, 29, 38, 10, 58, 26, 126, 17, 54, 35, 55, 35, 37, 110, 28, 41, 31, 45, 41, 20, 123, 17, 136, 41, 59, 51, 18, 14, 98, 35, 143]}, 'sampler_perf': {'mean_raw_obs_processing_ms': 0.1494050133826096, 'mean_inference_ms': 0.3030270243531414, 'mean_action_processing_ms': 0.04484232997177898, 'mean_env_wait_ms': 0.020527675791925257, 'mean_env_render_ms': 0.0}, 'num_faulty_episodes': 0, 'connector_metrics': {'ObsPreprocessorConnector_ms': 0.0020761489868164062, 'StateBufferConnector_ms': 0.0016286373138427734, 'ViewRequirementAgentConnector_ms': 0.0386197566986084}}	7.14829	3.57113	7.14829	{'training_iteration_time_ms': 3571.167, 'sample_time_ms': 1119.964, 'load_time_ms': 0.253, 'load_throughput': 16189340.451, 'learn_time_ms': 2449.34, 'learn_throughput': 1669.634, 'synch_weights_time_ms': 1.166}	1680195666	8179	2	6c451_00000

(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82020) 2023-03-30 10:01:03,714	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write]
(pid=82019) Resource usage vs limits 0:   0%|                                                                                                                                                      | 0/1 [00:00<?, ?it/s]
(pid=82019) Repartition 1:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]

(pid=82019) Repartition 2:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]

(pid=82019)   *- Repartition 2:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]


(pid=82019) Write 3:   0%|                                                                                                                                                                         | 0/1 [00:00<?, ?it/s]



(pid=82019) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82019) output: 0 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]
(pid=82019) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1:   0%|                                                                                                                    | 0/1 [00:00<?, ?it/s]


(pid=82019) Write: 0 active, 0 queued, 0.0 MiB objects 3:   0%|                                                                                                                                    | 0/1 [00:00<?, ?it/s]
(pid=82019) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.34 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                    | 0/1 [00:00<?, ?it/s]
(pid=82019) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 133.01it/s]


(pid=82019) Write: 1 active, 0 queued, 0.34 MiB objects 3:   0%|                                                                                                                                   | 0/1 [00:00<?, ?it/s]




(pid=82020) Resource usage vs limits 0:   0%|                                                                                                                                                      | 0/1 [00:00<?, ?it/s]





(pid=82020) Repartition 1:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]






(pid=82020) Repartition 2:   0%|                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]






(pid=82020)   *- Repartition 2:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]







(pid=82020) Write 3:   0%|                                                                                                                                                                         | 0/1 [00:00<?, ?it/s]








(pid=82020) Output 4:   0%|                                                                                                                                                                        | 0/1 [00:00<?, ?it/s]




(pid=82020) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]








(pid=82020) output: 0 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]





(pid=82020) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1:   0%|                                                                                                                    | 0/1 [00:00<?, ?it/s]







(pid=82020) Write: 0 active, 0 queued, 0.0 MiB objects 3:   0%|                                                                                                                                    | 0/1 [00:00<?, ?it/s]





(pid=82020) Repartition: 0 active, 0 queued, 0.0 MiB objects, 1 output 1:   0%|                                                                                                                    | 0/1 [00:00<?, ?it/s]




(pid=82020) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.33 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                    | 0/1 [00:00<?, ?it/s]





(pid=82020) Repartition: 0 active, 0 queued, 0.0 MiB objects, 0 output 1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 99.91it/s]







(pid=82020) Write: 1 active, 0 queued, 0.33 MiB objects 3:   0%|                                                                                                                                   | 0/1 [00:00<?, ?it/s]




(pid=82020) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]








(pid=82020) output: 1 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]







(pid=82020) Write: 0 active, 0 queued, 0.0 MiB objects 3: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 100.37it/s]




                                                                                                                                                                                                                         





                                                                                                                                                                                                                         






                                                                                                                                                                                                                         







                                                                                                                                                                                                                         








                                                                                                                                                                                                                         


(pid=82019) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82019) output: 1 queued 4:   0%|                                                                                                                                                              | 0/1 [00:00<?, ?it/s]


(pid=82019) Write: 0 active, 0 queued, 0.0 MiB objects 3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  9.73it/s]



                                                                                                                                                                                                                         0,  9.57it/s]
                                                                                                                                                                                                                         

                                                                                                                                                                                                                         


                                                                                                                                                                                                                         



                                                                                                                                                                                                                         2023-03-30 10:01:06,809	INFO tune.py:945 -- Total run time: 16.73 seconds (16.14 seconds for the tuning loop).

Then, we run training:

In [11]:

result = train_rl_bc_offline(path=path, num_workers=2, use_gpu=False)

Starting offline training

Tune Status

Current time:	2023-03-30 10:02:06
Running for:	00:00:14.18
Memory:	18.3/32.0 GiB

System Info

Using FIFO scheduling algorithm.
Logical resource usage: 0/8 CPUs, 0/0 GPUs

Trial Status

Trial name	status	loc	iter	total time (s)	ts	reward	episode_reward_max	episode_reward_min	episode_len_mean
AIRBC_914d8_00000	TERMINATED	127.0.0.1:82274	5	0.829429	20384	nan	nan	nan	nan

(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:18: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   DESCRIPTOR = _descriptor.FileDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:36: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:43: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:29: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _TENSORSHAPEPROTO_DIM = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:73: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:80: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:66: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _TENSORSHAPEPROTO = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:19: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   DESCRIPTOR = _descriptor.FileDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:33: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:37: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:41: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:45: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:49: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:53: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:57: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:61: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:65: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:69: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:73: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:77: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:81: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:85: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:89: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:93: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:97: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:101: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:105: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:109: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:113: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:117: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:121: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:125: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:129: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:133: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:137: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:141: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:145: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:149: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:153: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:157: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:161: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:165: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:169: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:173: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:177: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:181: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:185: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:189: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:193: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:197: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:201: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:205: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:209: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:213: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:217: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.EnumValueDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:27: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _DATATYPE = _descriptor.EnumDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:287: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:280: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _SERIALIZEDDTYPE = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:20: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   DESCRIPTOR = _descriptor.FileDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:39: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:46: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:32: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _RESOURCEHANDLEPROTO_DTYPEANDSHAPE = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:76: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:83: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:90: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:97: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:104: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:111: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py:69: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _RESOURCEHANDLEPROTO = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   DESCRIPTOR = _descriptor.FileDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:40: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:47: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:54: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:61: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:68: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:75: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:89: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:96: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:103: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:110: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:117: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:124: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:131: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:138: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:145: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:152: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:33: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _TENSORPROTO = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:183: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:190: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:197: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:176: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _VARIANTTENSORDATAPROTO = _descriptor.Descriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   DESCRIPTOR = _descriptor.FileDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:40: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:47: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:54: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:61: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:68: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:75: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
(pid=82274)   _descriptor.FieldDescriptor(
(RolloutWorker pid=82020) 2023-03-30 10:01:03,685	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[Repartition] -> TaskPoolMapOperator[Write] [repeated 3x across cluster]
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:61: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
(pid=82274)   if (distutils.version.LooseVersion(tf.__version__) <
(pid=82274) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:62: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
(pid=82274)   distutils.version.LooseVersion(required_tensorflow_version)):
(pid=82274) WARNING:tensorflow:From /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py:561: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version.
(pid=82274) Instructions for updating:
(pid=82274) experimental_relax_shapes is deprecated, use reduce_retracing instead
(AIRBC pid=82274) 2023-03-30 10:01:56,521	WARNING algorithm_config.py:636 -- Cannot create BCConfig from given `config_dict`! Property __stdout_file__ not supported.
(AIRBC pid=82274) 2023-03-30 10:01:56,624	INFO algorithm.py:527 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(AIRBC pid=82274) 2023-03-30 10:01:56,630	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadJSON] -> AllToAllOperator[Repartition]
(pid=82274) Resource usage vs limits 0:   0%|                                                                                                                                                      | 0/1 [00:00<?, ?it/s]
(pid=82274) ReadJSON 1:   0%|                                                                                                                                                                      | 0/2 [00:00<?, ?it/s]

(pid=82274) Repartition 2:   0%|                                                                                                                                                                   | 0/2 [00:00<?, ?it/s]


(pid=82274) Repartition 3:   0%|                                                                                                                                                                   | 0/2 [00:00<?, ?it/s]


(pid=82274)   *- Repartition 3:   0%|                                                                                                                                                              | 0/2 [00:00<?, ?it/s]



(pid=82274) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82274) output: 0 queued 4:   0%|                                                                                                                                                              | 0/2 [00:00<?, ?it/s]
(pid=82274) ReadJSON: 2 active, 0 queued, 1.49 MiB objects 1:   0%|                                                                                                                                | 0/2 [00:00<?, ?it/s]

(pid=82274) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 1.3 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]
(pid=82274) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 0.0 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]

(pid=82274) Resource usage vs limits: 0.0/8.0 CPU, 0.0/0.0 GPU, 1.3 MiB/512.0 MiB object_store_memory 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]



(pid=82274) output: 2 queued 4:   0%|                                                                                                                                                              | 0/2 [00:00<?, ?it/s]

                                                                                                                                                                                                                         , 214.99it/s]
                                                                                                                                                                                                                         

                                                                                                                                                                                                                         


                                                                                                                                                                                                                         



                                                                                                                                                                                                                         (pid=82294) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 10x across cluster]
(pid=82294)   DESCRIPTOR = _descriptor.FileDescriptor( [repeated 10x across cluster]
(pid=82294) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 80x across cluster]
(pid=82294)   _descriptor.FieldDescriptor( [repeated 80x across cluster]
(pid=82294) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:176: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 14x across cluster]
(pid=82294)   _TENSORSHAPEPROTO_DIM = _descriptor.Descriptor( [repeated 2x across cluster]
(pid=82294)   _TENSORSHAPEPROTO = _descriptor.Descriptor( [repeated 2x across cluster]
(pid=82294) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:217: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 94x across cluster]
(pid=82294)   _descriptor.EnumValueDescriptor( [repeated 94x across cluster]
(pid=82294) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:27: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 2x across cluster]
(pid=82294)   _DATATYPE = _descriptor.EnumDescriptor( [repeated 2x across cluster]
(pid=82294)   _SERIALIZEDDTYPE = _descriptor.Descriptor( [repeated 2x across cluster]
(pid=82294)   _RESOURCEHANDLEPROTO_DTYPEANDSHAPE = _descriptor.Descriptor( [repeated 2x across cluster]
(pid=82294)   _RESOURCEHANDLEPROTO = _descriptor.Descriptor( [repeated 2x across cluster]
(pid=82294)   _TENSORPROTO = _descriptor.Descriptor( [repeated 2x across cluster]
(pid=82294)   _VARIANTTENSORDATAPROTO = _descriptor.Descriptor( [repeated 2x across cluster]

(RolloutWorker pid=82295) DatasetReader 2 has 2, samples.

(pid=82294) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py:62: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. [repeated 4x across cluster]
(pid=82294)   if (distutils.version.LooseVersion(tf.__version__) < [repeated 2x across cluster]
(pid=82294)   distutils.version.LooseVersion(required_tensorflow_version)): [repeated 2x across cluster]
(pid=82294) WARNING:tensorflow:From /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py:561: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version. [repeated 2x across cluster]
(pid=82294) Instructions for updating: [repeated 2x across cluster]
(pid=82294) experimental_relax_shapes is deprecated, use reduce_retracing instead [repeated 2x across cluster]
(AIRBC pid=82274) 2023-03-30 10:02:05,472	WARNING util.py:67 -- Install gputil for GPU system monitoring.
(RolloutWorker pid=82295) 2023-03-30 10:02:05,480	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(AIRBC pid=82274) 2023-03-30 10:02:05,550	WARNING deprecation.py:50 -- DeprecationWarning: `remote_workers()` has been deprecated. Accessing the list of remote workers directly through remote_workers() is strongly discouraged. Please try to use one of the foreach accessors that is fault tolerant.  This will raise an error in the future!

Trial Progress

Trial name	agent_timesteps_total	connector_metrics	counters	custom_metrics	date	done	episode_len_mean	episode_media	episode_reward_max	episode_reward_mean	episode_reward_min	episodes_this_iter	episodes_total	evaluation	hostname	info	iterations_since_restore	node_ip	num_agent_steps_sampled	num_agent_steps_trained	num_env_steps_sampled	num_env_steps_sampled_this_iter	num_env_steps_trained	num_env_steps_trained_this_iter	num_faulty_episodes	num_healthy_workers	num_in_flight_async_reqs	num_remote_worker_restarts	num_steps_trained_this_iter	perf	pid	policy_reward_max	policy_reward_mean	policy_reward_min	sampler_perf	sampler_results	time_since_restore	time_this_iter_s	time_total_s	timers	timestamp	timesteps_total	training_iteration	trial_id
AIRBC_914d8_00000	20384	{}	{'num_env_steps_sampled': 20384, 'num_env_steps_trained': 20384, 'num_agent_steps_sampled': 20384, 'num_agent_steps_trained': 20384}	{}	2023-03-30_10-02-06	True	nan	{}	nan	nan	nan	0	0	{'episode_reward_max': 76.0, 'episode_reward_min': 11.0, 'episode_reward_mean': 29.6, 'episode_len_mean': 29.6, 'episode_media': {}, 'episodes_this_iter': 10, 'policy_reward_min': {}, 'policy_reward_max': {}, 'policy_reward_mean': {}, 'custom_metrics': {}, 'hist_stats': {'episode_reward': [13.0, 13.0, 11.0, 76.0, 48.0, 36.0, 14.0, 34.0, 14.0, 37.0], 'episode_lengths': [13, 13, 11, 76, 48, 36, 14, 34, 14, 37]}, 'sampler_perf': {'mean_raw_obs_processing_ms': 0.12616764114882342, 'mean_inference_ms': 0.2590957092866947, 'mean_action_processing_ms': 0.04460762030537042, 'mean_env_wait_ms': 0.021067194550248115, 'mean_env_render_ms': 0.0}, 'num_faulty_episodes': 0, 'connector_metrics': {'ObsPreprocessorConnector_ms': 0.0019359588623046875, 'StateBufferConnector_ms': 0.0017905235290527344, 'ViewRequirementAgentConnector_ms': 0.03647565841674805}, 'num_agent_steps_sampled_this_iter': 296, 'num_env_steps_sampled_this_iter': 296, 'timesteps_this_iter': 296, 'num_healthy_workers': 1, 'num_in_flight_async_reqs': 0, 'num_remote_worker_restarts': 0}	avnishs-mbp-3.lan	{'learner': {'default_policy': {'learner_stats': {'allreduce_latency': 0.0, 'grad_gnorm': 0.0902162678539753, 'policy_loss': 0.6931167542934418, 'total_loss': 0.6931167542934418}, 'model': {}, 'custom_metrics': {}, 'num_agent_steps_trained': 2000.0, 'num_grad_updates_lifetime': 9.5, 'diff_num_grad_updates_vs_sampler_policy': 8.5}}, 'num_env_steps_sampled': 20384, 'num_env_steps_trained': 20384, 'num_agent_steps_sampled': 20384, 'num_agent_steps_trained': 20384}	5	127.0.0.1	20384	20384	20384	4026	20384	4026	0	2	0	0	4026	{'cpu_util_percent': 31.2, 'ram_util_percent': 57.0}	82274	{}	{}	{}	{}	{'episode_reward_max': nan, 'episode_reward_min': nan, 'episode_reward_mean': nan, 'episode_len_mean': nan, 'episode_media': {}, 'episodes_this_iter': 0, 'policy_reward_min': {}, 'policy_reward_max': {}, 'policy_reward_mean': {}, 'custom_metrics': {}, 'hist_stats': {'episode_reward': [], 'episode_lengths': []}, 'sampler_perf': {}, 'num_faulty_episodes': 0, 'connector_metrics': {}}	0.829429	0.184188	0.829429	{'training_iteration_time_ms': 45.667, 'sample_time_ms': 26.685, 'load_time_ms': 0.193, 'load_throughput': 21115508.208, 'learn_time_ms': 17.555, 'learn_throughput': 232232.678, 'synch_weights_time_ms': 1.169}	1680195726	20384	5	914d8_00000

(pid=82313) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:21: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 5x across cluster]
(pid=82313)   DESCRIPTOR = _descriptor.FileDescriptor( [repeated 5x across cluster]
(pid=82313) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 40x across cluster]
(pid=82313)   _descriptor.FieldDescriptor( [repeated 40x across cluster]
(pid=82313) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py:176: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 7x across cluster]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) /Users/avnish/miniforge3/envs/ray/lib/python3.8/site-packages/tensorflow/core/framework/types_pb2.py:217: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool. [repeated 47x across cluster]
(pid=82313)   _descriptor.EnumValueDescriptor( [repeated 47x across cluster]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
(pid=82313) 2023-03-30 10:02:05,854	INFO streaming_executor.py:83 -- Executing DAG InputDataBuffer[Input] -> AllToAllOperator[RandomShuffle]
2023-03-30 10:02:06,881	INFO tune.py:945 -- Total run time: 14.67 seconds (14.17 seconds for the tuning loop).

And then, using the obtained checkpoint, we evaluate the policy on a fresh environment:

In [12]:

num_eval_episodes = 3

rewards = evaluate_using_checkpoint(result.checkpoint, num_episodes=num_eval_episodes)
print(f"Average reward over {num_eval_episodes} episodes: " f"{np.mean(rewards)}")

2023-03-30 10:02:11,829	WARNING checkpoints.py:109 -- No `rllib_checkpoint.json` file found in checkpoint directory /var/folders/jr/6lgb7_ln64v1kppw9szl17rc0000gn/T/tmp5kq688t7! Trying to extract checkpoint info from other files found in that dir.
2023-03-30 10:02:11,841	INFO policy.py:1285 -- Policy (worker=local) running on CPU.
2023-03-30 10:02:11,841	INFO torch_policy_v2.py:110 -- Found 0 visible cuda devices.

Average reward over 3 episodes: 24.666666666666668

In [ ]: