Red5 supports HD video streaming. You will also need a good webcam and lots of light to avoid low framerates and noise in the HD picture.
If your webcam outputs good quality video and it's encoded properly through your encoding settings and you have a good upload connection to your media server and your users have a good download connection with your media server then the stream should have low latency and not be pixelated.
Do you estimate those 300 users will all watch 1 HD video stream broadcasted by 1 user in the chat? If so I believe your 1st concern should be bandwidth out of that VPS and the monthly bandwidth usage.