Not being able to see other drivers is usually due to incompatibility on some level between the drivers that can't see each other. It can usually be fixed by having all the drivers that can't see or can't be seen leave the room, reset some things, and re-enter the room. Sometimes it takes more than one try to get it right. Sometimes they need to enter in a different order.
I can't say what the exact cause is. It could be a lost packet somewhere along the way, it could be a product of overly high ping times between two users causing their connection to time out to each other, or it could be any number of other causes. I can say that it happens in both open lobbies and friends only rooms when using a non-fixed host. I can also say that the only time I have seen it happen in a fixed host room, it was an open lobby and there was one driver that couldn't see any other driver and all the other drivers couldn't see him. I don't know if that was the same issue just acting differently in a fixed host room or if it was something else all together. I haven't run in enough fixed host rooms to say that it doesn't happen in those rooms.
I can't speak for who the culprit might be in D6 but I think it's pretty common knowledge who the culprit is when it happens in D3 as that driver is almost always involved in not being able to see others or others seeing him. If you start to keep track of who is involved when it happens, you will probably be able to isolate the problem connection(s).