Meanwhile, Ray J has yet to weigh in on this situation